Versions Compared

Old Version 20

changes.mady.by.user Alessandra Filippi

Saved on

compared with

New Version 21

changes.mady.by.user Alessandra Filippi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  1. Production a Millepede input binary
  2. Run millepede using mille.bin creating a resulting file with correction to the constants, e.g. millepede.res
  3. Create a new compact detector description with the new corrections, e.g. compact_new.xml
    1. java -cp hps-distribution-3X.0Y.4Z-SNAPSHOT-bin.jar org/hps/svt/alignment/BuildMillepedeCompact -c compact.xml millepede.res -o compact_new.xml 
    2. Use option "-r" to replace instead of adding to existing constants in the compact.xml
  4. Create a new detector with this new compact_new.xml file.

Production of a Millepede binary input file

BEWARE: the following procedure (old python-based procedure) is valid only for hps-java versions < 4.2 (oct2018). From this version on, the binary file to be provided as input to Millepede is produced automatically upon the reconstruction, so steps 2-5 are skipped. Go to the section "New procedure" for updated instructions.

Old python-based procedure (hps-java < 4.2 oct18)

The Millepede input binary file is obtained through the following steps:

  1. Reconstruct real data with GBL (to this purpose, you need to include the GblOutputDriver in your steering file).

    1. For Montecarlo data:

      1. run the readoout with the steering file (for instance, for test run data: HPS2014ReadoutNoPileup.lcsim)
      2. run the recon with the steering file (for instance, for test run data: HPS2014OfflineNoPileUp.lcsim)
    2. For real data you can use whatever steering file (usually the most update version will work)  but remember to include the mentioned GblOutputDriver drivers as mentioned above, if they are not present (usually they are not, for production purposes):
      <driver name="GBLOutputDriver"/>
      ...something else in the lcsim file...
       <driver name="GBLOutputDriver" type="org.hps.recon.tracking.gbl.GBLOutputDriver">
               <debug>0</debug>
               <isMC>false</isMC>   <!-- use true for MC data here -->        
                <gblFileName>${outputFile}.gbl</gblFileName>
      </driver>
      Of course, for straight tracks use a proper steering file.
    3. For the purpose of alignment you are recommended to drop ghost hits, including in the HelicalHitDriver driver the following line:
      <rejectGhostHits>true</rejectGhostHits>

  2. Check that at the end of reconstruction a out.gbl ascii file (or, named as you decided in the GblOutputDriver) is produced.
  3. Remember that by default the geometry is taken from the database. If you want to force the use of your own geometry, you must provide it in the compact.xml file in a given detector. For MonteCarlo data, set the run number to zero during readout and reconstruction. This is done adding the flag -Drun=0 when running the readout. For real data, use the -DdisableSvtAlignmentConstants flag and a proper run number to select other calibration constants. IMPORTANT: remember to re-compile hps-java before running each time you change the compact.xml file! (this is the most common error).
  4. The out.gbl file is read by a python procedure, called gbltst-hps.py. You must download with git the current version of the software from the github repository as described in the following. This will create a hps-gbl directory. After having configured your account and username for git use, issue the following commands:
    (the second and third command need to be issued just upon installation, and they are needed since some directories with utility files are shared with other software packages which will be described later).
    Once you have downloaded the code, remember to install the GBL software, if you already haven't it. In a directory parallel to hps-gbl download the GBL software using svn:
    • svn co http://svnsrv.desy.de/public/GeneralBrokenLines/tags/v01-16-04 GeneralBrokenLines
      (or check the newest release, and get it). To compile it:
    • cd GeneralBrokenLines/cpp
    • mkdir build; cd build
    • cmake ../
    • make
    • make install
    • make doc (if you want it)
      Note: if you have installed the latest cmake version, probably it won't compile. You must prevent the compilation to search for C++11 support (the default for newest cmake). To do this, you have to set as compilation flag -std=c++0x adding it to the c++ compilation line. Either you do it in the cmake configuration files, or (quickest) you add by hand this flag at the end of the CXX_FLAGS line, in these two files:
      • GeneralBrokenLines/cpp/build/CMakeFiles/GBL.dir/flags.make
      • GeneralBrokenLines/cpp/build/examples/CMakeFiles/GBLpp.dir/flags.make
  5. The gbl python procedure reads the out.gbl file and prepares the binary read by Millepede. You must run python from the hps-gbl directory. This is the shortest syntax (-h shows all possible options):
    • python gbltst-hps.py $GBLFILE --name $OUTNAME --ntracks $MAXTRKS --nopause --save
      You can provide the input gbl filename, the output file name and the number of tracks to be processed as logical names in a shell script.
      A heap of pdf files are produced containing plots of several quantities for top/bottom halves, with long names that should be self-explaining (but at the moment they are not and they are too long, this needs to be improved). You will also file a .root file containing the single root histograms, and a .ps file containing a summary of the plots ready to be printed.
      The file gbltst-hps.py contains the instructions to extract the useful information on tracks and hits from the ascii file and write the input file for Millepede; it reruns GBL on the tracks. If you want to add/change some of the output plots/histograms, you have to modify both the gbl_plots.py file (in which they have to be booked) and the gbltst-hps.py file, in which they have to be filled.
      See the help (or, better, the code) for indication to further functionalities:
      •  python gbltst-hps.py --help
      Use the --batch flag if you not running interactively (if you are submitting the job on a batch system, for instance) or you do not want the output histograms to pop out when the elaboration is finished (they can be annoyingly invasive on your screen for about a minute).
      At the end of python processing, you should also find a MilleBinaryISN.dat file (the name could slightly change), which is the input file to be read by Millepede.
      Keep in mind that this procedure is very RAM consuming (all track informations are stored in memory once read from the GBL ascii file). Therefore, in general it is not a good idea to run many multiple sessions, or even single sessions with many events, on the same processor, unless you want to risk your system be stuck. 
      Note on root compilation: root must be compiled including the python support, otherwise python stops with an error complaining about root libraries missing. A good practice is to put in your profile and instruction to run automatically $ROOTSYS/bin/thisroot.(c)sh, which provides the correct root-python environment and libraries for your system.
      Note on straight tracks processing. If you want to process straight tracks (no magnetic field) you need to used a slightly different python script:
      • python gbl-hps-ST.py  $GBLFILE --save --nopause
      Of course, you will void histograms for quantities related to helical track parameters, impact parameters, momenta etc.
      Note on truth MC information: for the moment being, if the GBL procedure is being run on Montecarlo data (flag --mc), no output is extracted for MC truth quantities. This needs to be fixed (it was working up to a certain hps-java version, need to be checked if it is a problem of info extraction from the banks by the python script, or it is something more related to the way the information is stored in the readout/reconstruction of the simulated data).
  6. Once you run GBL on the out.gbl file, you can check the quality of the alignment/geometry studying residuals, kinks, pulls and several other useful distributions. All histograms are contained in a file named gbl_$OUTNAME_*.root (the full name can be very long and depends on the used flags). A set of useful root macros can be found here:The master branch contains macros which are compliant to root v5.34, the branch root6 contains macros working with root6 as well (this is the development branch). In the git bundle you can find a README file with some indications about how to run the macros, and what they do.
    Note: due to a reversed sign in the magnetic field coordinates in the GBL procedure, the charge of tracks in the produced rootfile is reversed as well (keep this in mind when analysing this output). In many cases, also the u-v axes have to be flipped to provide the correct orientation of the sensors in the space (this is already done almost everywhere in the macros). By inspecting the trend of residuals and distributions one can judge which are the most critical degrees of freedom to float in the following Millepede iteration.

 

New procedure (hps-java > 4.1 oct18)

The Millepede input binary file is obtained directly upon running hps-java provided in the steering file the GblOutputDriver driver is called, and some options are added to  GblRefitterDriver. The following procedure must be followed:

  1. Include GblOutputDriver in your steering file and add the following options to GblRefitterDriver:
    1. <driver name="GBLOutputDriver"/>
    2. <driver name="GBLOutputDriver" type="org.hps.recon.tracking.gbl.GBLOutputDriver">
    3. <driver name="GBLRefitterDriver" type="org.hps.recon.tracking.gbl.GBLRefitterDriver">

              <storeTrackStates>true</storeTrackStates>

               <writeMilleBinary>true</writeMilleBinary>

                <milleBinaryFileName>millepedeData.bin</milleBinaryFileName>

       </driver>

    in this way the output binary file to be read by Millepede will be called millepedeData.bin. A further root ouput file, named after -Doutputfile (outputfile.root, the default being GBLplots.root) will be produced, containing residuals and several distributions to be analyzed to judge the alignment quality. See point 3.
  2. Check that at the end of reconstruction the files millepedeData.bin and outputFile.root (GBLplots.root) are produced.
  3. To check the quality of the alignment/geometry studying residuals, kinks, pulls and several other useful distributions. A set of useful root macros can be found in github https://github.com/afilippi67/DataQualityMacros: the relevant suite to analyze the new histograms (which have different namings as compared to the "old" procedure) is in branch root6_newReco (the branch root6 is for the "old" procedure).In the git bundle you can find a README file with some indications about how to run the macros, and what they do. Note that some histograms, if compared to "old" procedure, can be missing (in particular, those related to track momenta or output helix parameters).

Running Millepede

  1. Once you have the binary file, millepede is ready run from the hps-mille directory. This directory is setup downloading the Millepede software by github using the following commandRemember to compile the fortran sources of the MillepedeII software. It comes with the git bundle (or you can download it from https://www.wiki.terascale.de/index.php/Millepede_II) but you have to compile it in your system beforehand (note: with gfortran you might have to slightly modify the Makefile by hand, because the code version is frozen and not aligned anymore to more modern gfortran versions/libraries).
  2. To run millepede use the following commands:
    • cd hps-mille
    • ./runMP.py -i../hps-gbl/milleBinaryISN.dat -M NAMES
    where NAMES is a list of parameters coded via the following regexp: L(1-6)[AS]?[hs]?[tb]_([tr])([uvw]) having the following meaning:
    1. 1-6: layer number
    2. A: axial, S: stereo
    3. h: hole, s: slot
    4. t: top, b:bottom
    5. _t: translation, _r: rotation
    6. u, v, w: coordinates on the sensor reference system
    if some of the parameters preceded by "?" are omitted, both the choices are selected
    Millepede produces as output, among several files, the millepede.res file which contains the corrections found by Millepede for the floated parameters.
    To keep track of the subsequent iterations, it is wiser to store them file that can be read automatically by the runMP.py procedure. The file is called floatoptions.py and it contains a series of "options", each of them containing a list of sensors to be floated in the same or multiple iterations, coded following the regexp pattern described above. The user can add as many options as needed, ordering them by consecutive numbers which are then used to label a "switch number" used to select the needed option upon running the procedure. The corresponding command to be issued is:
    • ./runMP_batch.py --run --files milleBinaryISN.dat -s switchNumber -b
    The output file will be labeled with the title string provided in the option definition and the sequence of desired floating operations (keep it simple because when its gets too long -usually for several iterations in one go or multiple sensor floatings- the procedure breaks). Usually a run with Millepede takes some minutes on about 1M tracks, with a reasonable amount of floated parameters, so it is not a time consuming step.
    Note on binary input files. Millepede can run on multiple input files, but they will be processed one at a time (and the result will be provided separately for each one), which in general is not what you want when you intend to feed it with cumulated statistics (keep in mind that it can only process 1.5M tracks in total, so keep under control the size of your samples). What you need to do, for instance when you want to provide a mixed sample of curved+straight tracks, is to use a single input file which merges several other partial binary files. To produce it you just need to concatenate the single files, like:
    • cat mille_straight1.dat mille_curved1.dat mille_straight2.dat ... > mille_mixed.dat
    When merging curved and straight track sets together, try to sort them evenly in order to get an homogeneous composition of the final sample.  

Create a new compact based on Millepede corrections

The new geometry file is produced in the hps-mille directory. Use the following python wrapper that reads the millepede.res file and writes the compact_new.xml file (it replaces the procedure describes in point 3a of the introductory notes in this section):

...

ATTENTION: all rotation offsets have to be summed, which means their sign doesn't have to be flipped when building the new compact.xml file. In some cases two "-" signs appear the the xml file: don't worry, this behaves as expected in the  java code.

 

Create a new detector based on a new compact.xml file

The instruction on how to create a new detector given your new .xml geometry file can be found here: Detector Geometry Overview, section: Adding A New Detector. Remember to rename the compact_new.xml file into compact.xml, write a few lines of clever/useful comment in the .xml file to remember which kind of geometry this is, and recompile hps-java before running with the new geometry. If you want to to process straight tracks with the new geometry, remember to modify properly the compact.xml file setting the constBFieldY variable to zero, and commenting out the magnetic field map file definition in <fields> while activating the constant BoxDipole field.

...

That's it! If you got to this point, either you already find a perfect alignment or you are ready to restart the full procedure from point 1 of the list at the beginning of this section, for an uncountable number of iterations.

 

...

 

Note on beam spot integration in Millepede

BEWARE: not recommended for beginners: at your risk! note that the procedure for the moment being does not provide satisfactory results, so this paragraph is kept for reference purpose mainly)

...