...

Application panel under ROI (Region Of Interest) tab contains controlls re-implemented algorithms of the command line roicon, described in Make ndarray for ROI mask.

...

Geometry

  Image Added

Metrology

Metrology tool processes xlsx files from opitcal measurements, produces calibration constants of types geometry, center and tilt, and deploy them in calibration directory.

xlsx files are human-made, so, one has to be careful and check that the text file, produced in auto-conversion by command #2 has correct format.

 anchor

Alignment

Image Added

Alignment tool works with geometry files, produced in Metrology tool and allows to tune quad or other sensor positions in the detector, using image with real objects (rings, shadows, etc).

More details about this tool can be found in Detector Alignment Tool.

Configuration

Configuration tool has its own tab bar switching between panels with sets of parameters.

...

• Dir work button - launch directory selection window for work directory. Selected directory shows up in the next right information field.
• Dir results button - launch directory selection window for results directory. Selected directory shows up in the next right information field. This directory is not currently in use.
• File prefix editable field - sets common prefix of all files created in the work directory, excluding the log file.
• Queue combo box - sets the queue name for batch jobs from available list containing psnehq, psfehq, etc. 
• Event start/end editable fields - set the first and last event in current run for calibration processing job.
• Threshold RMS, ADU (editable field) - sets the threshold on RMS in ADU (Analog to Digit Units) for hot pixel mask. If RMS for pixel exceeds this threshold - pixel is marked as hot in the hot pixel mask file. "Zero" is a special value, indicating that no pixels will be flagged as hot pixels because of noise.
• Threshold MIN/MAX (editable fields) - if a pixel's pedestal is outside the MIN/MAX range it will be flagged in the hot pixel mask file.
• Deploy hotpix mask check box - if checked - hotpixel mask file will be deployed along with pedestal file at click on Deploy button in the List of run items.
  
  

...

Project status

Current 2015-04-08 status corresponds to:

relinfo
Pkg/Rel        TEST    BASE(ana-0.1113.018)
CalibManager   HEAD   V00-00-41-83
ImgAlgos       HEAD   V00-03-80
PSCalib        HEAD   V00-02-33
PSQt           HEAD   -

Completely implemented applications under calibman tabs:

• Dark
• ROI 
• File Manager
  • Single File Manager
  • Group File Manager
• Configuration 
• Geometry

  Partly implemented calibman tab(s):

  • Geometry
    • Metrology

  Newly implemented:

  • • Metrology - implemented for CSPAD CXI-V1, XPP-V2, CSPAD2x2
    • Alignment 

  calibrun - command-line application is fully functional

  Recommended updates

  ...

  On 20142015-04-29

  Use default version of all involved packages.

  Until 15: ana-current is not pointing to (a.k.a. ana-0.11.0 use command

   

  sit_setup ana-0.11.0

   14.01) does not need in updates.

   

   

  How to get and run calibman

  ...

  Code Block
  cd <your-favorite-directory>; sit_setup ana-0.11.0; calibman;

   

  Get latest version, compile, and run

  The latest version of this application with recent modifications for packages can be created and executed from the local release directory. Login to psana, then run use commands with appropriate substitutions for <...> fields:

  Code Block
  cd <your-favorite-test-directory>; newrel ana-current test-CalibManager; cd test-CalibManager; sit_setup <release-tag>; addpkg CalibManager HEAD; addpkg <package2> HEAD; addpkg <package3> HEAD; ... scons; calibman;

   

   

  First session

  No matter what tool tab is opening by default in the first session, it would be better to make sure that configuration parameters are set correctly. To do that click on tab Configuration, which opens Configuration Settings panel,  then click on tab Parameters. Most parameters can be used with default values, but it is important to check, that
  

  • correct queue will be used for batch jobs. If necessary, change Queue in combo box.

  • correct range of events is set for event processing.

  • Threshold on rms is good for calibrated detector(s). Evaluation of this threshold requires in simple analysis which can be done after the 1st iteration on event averaging. Files containing averaged intensities and their rms values can be plotted as image and spectra. Spectrum of rms gives a rough idea of how to set this threshold. When threshold is set, second iteration on event averaging will produce correct mask-file for hot pixels, which rms exceeds the threshold.

  How to calibrate / deploy files for dark correction

  • Click on tab Dark.
  • Set Instrument, Experiment, from combo boxes and click on Select button to select Detector(s) from pop-up window.
  • Set filter for runs using combo boxes to the right from labels Show runs and for detectors.
  • For runs in the table, which you want to calibrate, click on button Go, relax and wait until button Deploy show-up to the right from button. A few buttons Go can be clicked sequentially for different runs in order to run appropriate jobs in batch simultaneously.
  • Change validity range if necessary, click on Deploy button.  If everything is ok, it is done!

  Known problems and ways around

  World is not perfect. Anything may happen during this long workflow. Expert mode sheds light on most of the problems. Here is a list of most frequent problems;

  • AFS token is expired for your terminal session. Batch system will complain. Before launching calibman application, first execute kinit and  aklog commands.
  • job on data from NEH/FEH are submitted to the wrong butch queue psfehq/psnehq. Set appropriate queue.
  • DB has a list of sources (detectors) which does not coincide with what is written in xtc file. After job is submitted, check the list of detectors found in xtc scan and check only available detectors in the Detector selection menu.

  Command line interface

  Available since release ana-0.10.16  (CalibManager V00-00-40)

  Command line interface for dark-run calibration of detectors is implemented by P.O.C.s request. It can be started by the command calibrun with parameters.  The list of parameters can be dumped on terminal by the command:

  Code Block
  calibrun -h

  which prints

  First session

  No matter what tool tab is opening by default in the first session, it would be better to make sure that configuration parameters are set correctly. To do that click on tab Configuration, which opens Configuration Settings panel,  then click on tab Parameters. Most parameters can be used with default values, but it is important to check, that
  

  • correct queue will be used for batch jobs. If necessary, change Queue in combo box.

  • correct range of events is set for event processing.

  • Threshold on rms is good for calibrated detector(s). Evaluation of this threshold requires in simple analysis which can be done after the 1st iteration on event averaging. Files containing averaged intensities and their rms values can be plotted as image and spectra. Spectrum of rms gives a rough idea of how to set this threshold. When threshold is set, second iteration on event averaging will produce correct mask-file for hot pixels, which rms exceeds the threshold.

  How to calibrate / deploy files for dark correction

  • Click on tab Dark.
  • Set Instrument, Experiment, from combo boxes and click on Select button to select Detector(s) from pop-up window.
  • Set filter for runs using combo boxes to the right from labels Show runs and for detectors.
  • For runs in the table, which you want to calibrate, click on button Go, relax and wait until button Deploy show-up to the right from button. A few buttons Go can be clicked sequentially for different runs in order to run appropriate jobs in batch simultaneously.
  • Change validity range if necessary, click on Deploy button.  If everything is ok, it is done!

  Known problems and ways around

  World is not perfect. Anything may happen during this long workflow. Expert mode sheds light on most of the problems. Here is a list of most frequent problems;

  • AFS token is expired for your terminal session. Batch system will complain. Before launching calibman application, first execute kinit and  aklog commands.
  • job on data from NEH/FEH are submitted to the wrong butch queue psfehq/psnehq. Set appropriate queue.
  • DB has a list of sources (detectors) which does not coincide with what is written in xtc file. After job is submitted, check the list of detectors found in xtc scan and check only available detectors in the Detector selection menu.

  Command line interface

  Available since release ana-0.10.16.


  Command line interface for dark-run calibration of detectors is implemented by P.O.C.s request. It can be started by the command calibrun with parameters.  The list of parameters can be dumped on terminal by the command:

  Code Block
  calibrun -h

  which prints

  Code Block
  Usage: calibrun [options] args Example: calibrun -e xppi0613 -d CSPAD,OPAL1000 -c ./calib -P -D -r 173 calibrun - command line calibration utility Options: -h, --help show this help message and exit -e EXP, --exp=EXP experiment name, ex.: cxi12345 -r RUNNUM, --run=RUNNUM run number -n NUM_EVENTS, --num_events=NUM_EVENTS number of events to process -s SKIP_EVENTS, --skip_events=SKIP_EVENTS number of events to skip before processing -m SCAN_EVENTS, --scan_events=SCAN_EVENTS number of events to scan data and search for selected detector(s) -f EVENT_CODE, --event_code=EVENT_CODE
  Code Block
  Usage: calibrun [options] args Example: calibrun -e xppi0613 -d CSPAD,OPAL1000 -c ./calib -P -D -r 173 calibrun - command line calibration utility Options: -h, --help show this help message and exit -e EXP, --exp=EXP event code experimentfor name, ex.: cxi12345dark event selection -rt RUNNUMTHR_RMS, --run=RUNNUM thr_rms=THR_RMS threshold on run numberrms for hot pixel mask evaluation -nv NUM_EVENTSRUNRANGE, --num_events=NUM_EVENTSrunrange=RUNRANGE number of events to processvalidity run range, ex.: 123-567 or 123-end -sq SKIP_EVENTSQUEUE, --skip_events=SKIP_EVENTSqueue=QUEUE numberqueue ofname, eventsex: topsnehq, skippsfehq, beforepsanacsq. processing -t THR_RMS, --thr_rms=THR_RMSIf specified threshold- on rms for hot pixel mask evaluationrun in batch -vw RUNRANGEWORKDIR, --runrangeworkdir=RUNRANGEWORKDIR validitywork run rangedirectory, ex.: 123-567 or 123-end./work -qc QUEUECALIBDIR, --queuecalibdir=QUEUECALIBDIR queue namenon-standard calibration directory, ex: psnehq, psfehq, psanacsq. If specified./calib -x XTCDIR, --xtcdir=XTCDIR - run in batch -w WORKDIR, --workdir=WORKDIRnon-standard path to xtc directory(<INS>/<EXP>/xtc are work directoryrequired), ex: ./work -c CALIBDIR, --calibdir=CALIBDIRmydir/xpp/xppi0613/xtc, or non-standard calibration directory, ex: ./calib/reg/d/ffb/cxi/cxi12345/xtc -d DETECTOR, --detector=DETECTOR detector names separated by comma, ex:CSPAD,CSPAD2x2,PNCCD -p PRINT_BITS, --print_bits=PRINT_BITS 0-print nothing, +1-debug, +2-info, +4-warning, +8-error, +16-critical; default=036, minimal recommended=020 -P, --process process xtc files and produce +8-error, +16-critical; default=036, minimalcalib files under the recommended=020"work" directory -PD, --processdeploy processdeploy xtccalibrated files andunder producethe "calib" files under thedirectory -L, --loadcfg load and use configuration parameters from file after "work" directory -D, --deploy deploy calibrated files under the "calibcalibman" directory

  The calibrun application reuses code and approach developed for calibman. At start calibrun loads parameters from configuration file confpars-calibman.txt.  By default this file is the same configuration file as for calibman. If parameters were already set in  confpars-calibman.txt, then calibrun can be started with a single parameter - run number (-r <run-number>) and options for processing (-P) and deployment (-D), for example:

  ...

  By default the calibrun command produces quite extensive output on terminal which in most cases is sufficient to figure out what is going on.  If the print-control option (-p  0) is provided, the major output will be suppressed and only begin, end of job comments, and warning messages will be printed. Most complete output can be found in the session  log file auto-saved in the work directory with relevant time stamp in the name, for example: ./work/2014-04-16-16:43:52-log.txt

  GUI-based calibration manager calibman has the same file name conventions and can always be used to inspect all output files and problems generated by the command line application  calibrun.

  References

  example: ./work/2014-04-16-16:43:52-log.txt

  GUI-based calibration manager calibman has the same file name conventions and can always be used to inspect all output files and problems generated by the command line application  calibrun.

  How to run on ffb data

  Command calibrun can work on ffb data using option -x <path-to-xtc>. This option sets non-standard path to xtc files. For example:

  Code Block
  calibrun <options> -x /reg/d/ffb/cxi/cxi12345/xtc <more-options>

  The "live" mode is turning on automatically if the directory path has "ffb" sub-folder.

  References

  • Calibration Store
  • Calibration management tool developement notes
  • Description of Calibration database
  • Mask Editor 
  • Make ndarray for ROI mask
  • Detector Alignment Tool
  • Detector Geometry