Page History
...
Application panel under ROI (Region Of Interest) tab has a minimal number of control fields, which allow to specify the file with image and open/close Mask Editor UI in separate window.
When launched, Mask Editor works as completely independent application. The Mask Editor allows to draw on image any number of a few geometry figure types and finally generate and save positive or negative mask for loaded image.
...
controlls re-implemented algorithms of the command line roicon
, described in Make ndarray for ROI mask.
The files with ndarray and geometry constants should be selected as input parameters in the first two options of this GUI. The names of other intermediate and output files for "Image", "Mask image", and "Mask ndarray" are optional. Next three buttons
- 3. reconstruct image from ndarray,
- 4. Open Mask Editor, and
- 5. Convert mask image to ndarray
executes associated procedures. "Mask control bits" parameter controls sensor pixels, which will to be masked. For example, for CSPAD
- Mask control bits = 0 - do not mask sensor pixels
- +1-mask edges,
- +2-wide pixels,
- +4-non-bounded pixels,
- +8-non-bounded pixel neighbours.
When launched, Mask Editor works as completely independent application. The Mask Editor allows to draw on image any number of a few geometry figure types and finally generate and save positive or negative mask for loaded image. When mask is created it needs to be saved in the "Mask image" file. After that mask editor can be closed.
All files and images of the ROI application can be "Plot"-ted and "View"-ed. For now, mask file is saved in the work directory and is not deployed. In stead, its path could be used in the ImgAlgos.NDArrCalib module.
Anchor | ||||
---|---|---|---|---|
|
Single File Manager
File manager tool consists of Status panel and control panel:
File manager control panel contains a set of control fields with functionality as follows.
- Button Browse - launches the file selection window which helps to select any calibration file in any location. Location is hinted in the intermediate pop-up window. Selected file is displayed in the next-left information field.
- View button - opens/closes the file viewer in separate window.
- Plot buttons - opens/closes the plotter window and plots the file as 2-d image (if applicable).
- Delete button - deletes the selected file from disc.
- Move/Copy buttons - perform associated actions on selected file. The destination directory and file name is defined by the Instrument/Experiment/Detector selection panel and next-right filelds:
- for detector selection button - sets the destination detector name from drop down menu,
- calibration type selection button - sets the destination calibration type, and
- two editable fields - set the run range for destination calibrationfile.
Anchor | ||||
---|---|---|---|---|
|
Group File Manager
Anchor | ||||
---|---|---|---|---|
|
Geometry
Anchor | ||||
---|---|---|---|---|
|
Metrology
Metrology tool processes xlsx
files from opitcal measurements, produces calibration constants of types geometry
, center
and tilt
, and deploy them in calibration directory.
xlsx
files are human-made, so, one has to be careful and check that the text file, produced in auto-conversion by command #2 has correct format.
Alignment
Alignment tool works with geometry
files, produced in Metrology tool and allows to tune quad or other sensor positions in the detector, using image with real objects (rings, shadows, etc).
More details about this tool can be found in Detector Alignment Tool.
Anchor | ||||
---|---|---|---|---|
|
Configuration
Configuration tool has its own tab bar switching between panels with sets of parameters.
Two panels are implemented currently:
Anchor | ||||
---|---|---|---|---|
|
Configuration Parameters
This panel has a few active fields:
- Dir work button - launch directory selection window for
work
directory. Selected directory shows up in the next right information field. - Dir results button - launch directory selection window for
results
directory. Selected directory shows up in the next right information field. This directory is not currently in use. - File prefix editable field - sets common prefix of all files created in the work directory, excluding the log file.
- Queue combo box - sets the queue
...
Single File Manager
File manager tool consists of Status panel and control panel:
File manager control panel contains a set of control fields with functionality as follows.
- Button Browse - launches the file selection window which helps to select any calibration file in any location. Location is hinted in the intermediate pop-up window. Selected file is displayed in the next-left information field.
- View button - opens/closes the file viewer in separate window.
- Plot buttons - opens/closes the plotter window and plots the file as 2-d image (if applicable).
- Delete button - deletes the selected file from disc.
- Move/Copy buttons - perform associated actions on selected file. The destination directory and file name is defined by the Instrument/Experiment/Detector selection panel and next-right filelds:
- for detector selection button - sets the destination detector name from drop down menu,
- calibration type selection button - sets the destination calibration type, and
- two editable fields - set the run range for destination calibrationfile.
...
Group File Manager
...
Geometry
...
Metrology
...
Configuration
Configuration tool has its own tab bar switching between panels with sets of parameters.
Two panels are implemented currently:
...
Configuration Parameters
This panel has a few active fields:
- Dir work button - launch directory selection window for
work
directory. Selected directory shows up in the next right information field. - Dir results button - launch directory selection window for
results
directory. Selected directory shows up in the next right information field. This directory is not currently in use. - File prefix editable field - sets common prefix of all files created in the work directory, excluding the log file.
- Queue combo box - sets the queue name for batch jobs from available list containing psnehq, psfehq, etc.
- Event start/end editable fields - set the first and last event in current run for calibration processing job.
- Threshold RMS, ADU (editable field) - sets the threshold on RMS in ADU (Analog to Digit Units) for hot pixel mask. If RMS for pixel exceeds this threshold - pixel is marked as hot in the hot pixel mask file. "Zero" is a special value, indicating that no pixels will be flagged as hot pixels because of noise.
- Threshold MIN/MAX (editable fields) - if a pixel's pedestal is outside the MIN/MAX range it will be flagged in the hot pixel mask file.
- Deploy hotpix mask check box Deploy hotpix mask check box - if checked - hotpixel mask file will be deployed along with pedestal file at click on Deploy button in the List of run items.
...
Note |
---|
Modal dialog assumes that one of the buttons in this type of dialog window need to be clicked, otherwise UI will be frozen. |
Project status
Current 2015-04-08 status corresponds to:
Code Block |
---|
relinfo Pkg/Rel TEST BASE(ana-0.1013.1518) CalibManager HEAD V00-00-2883 ImgAlgos HEAD V00-03-80 PSCalib HEAD V00-02-33 PSQt HEAD - |
Completely implemented applications under calibman
tabs:
- Dark
- ROI
- File Manager
- Single File Manager
- Group File Manager
- Configuration
Partly implemented tab(s):
- Geometry
- Metrology
Recommended updates for release ana-0.10.15
On 2014-04-17
- all currently known problems has been fixed,
- command-line interface is available
Code Block |
---|
addpkg CalibManager HEAD; (V00-00-37) |
- - implemented for CSPAD CXI-V1, XPP-V2, CSPAD2x2
- Alignment
calibrun
- command-line application is fully functional
Recommended updates
On 2015-04-15: ana-current (a.k.a. ana-0.14.01) does not need in updates.
How to get and run calibman
...
Code Block |
---|
cd <your-favorite-directory>;
sit_setup;
calibman; |
Get latest version, compile, and run
The latest version of this application with recent modifications for packages can be created and executed from the local release directory. Login to psana, then run use commands with appropriate substitutions for <...> fields:
Code Block |
---|
cd <your-favorite-test-directory>;
newrel ana-current test-CalibManager;
cd test-CalibManager;
sit_setup;
addpkg CalibManager HEAD;
addpkg <package2> HEAD;
addpkg <package3> HEAD;
...
scons;
calibman; |
Before starting calibman
If terminal session was open long time ago, it is good idea to update credentials prior launching the calibman application:
Code Block |
---|
kinit;
aklog; |
First session
No matter what tool tab is opening by default in the first session, it would be better to make sure that configuration parameters are set correctly. To do that click on tab Configuration, which opens Configuration Settings panel, then click on tab Parameters. Most parameters can be used with default values, but it is important to check, that
correct queue will be used for batch jobs. If necessary, change Queue in combo box.
correct range of events is set for event processing.
Threshold on rms is good for calibrated detector(s). Evaluation of this threshold requires in simple analysis which can be done after the 1st iteration on event averaging. Files containing averaged intensities and their rms values can be plotted as image and spectra. Spectrum of rms gives a rough idea of how to set this threshold. When threshold is set, second iteration on event averaging will produce correct mask-file for hot pixels, which rms exceeds the threshold.
How to calibrate / deploy files for dark correction
- Click on tab Dark.
- Set Instrument, Experiment, from combo boxes and click on Select button to select Detector(s) from pop-up window.
- Set filter for runs using combo boxes to the right from labels Show runs and for detectors.
- For runs in the table, which you want to calibrate, click on button Go, relax and wait until button Deploy show-up to the right from button. A few buttons Go can be clicked sequentially for different runs in order to run appropriate jobs in batch simultaneously.
- Change validity range if necessary, click on Deploy button. If everything is ok, it is done!
Known problems and ways around
World is not perfect. Anything may happen during this long workflow. Expert mode sheds light on most of the problems. Here is a list of most frequent problems;
- AFS token is expired for your terminal session. Batch system will complain. Before launching
calibman
application, first executekinit
andaklog
commands. - job on data from NEH/FEH are submitted to the wrong butch queue psfehq/psnehq. Set appropriate queue.
- DB has a list of sources (detectors) which does not coincide with what is written in xtc file. After job is submitted, check the list of detectors found in xtc scan and check only available detectors in the Detector selection menu.
Command line interface
Available since release ana-0.10.15, CalibManager V00-00-35
.
In addition to the GUI-based calibration, command line interface for dark-run calibration of detectors is implemented by POCs request. It can be started by the command calibrun
with parameters. The list of options/parameters can be dumped by the command:
Code Block |
---|
calibrun -h |
which prints
First session
No matter what tool tab is opening by default in the first session, it would be better to make sure that configuration parameters are set correctly. To do that click on tab Configuration, which opens Configuration Settings panel, then click on tab Parameters. Most parameters can be used with default values, but it is important to check, that
correct queue will be used for batch jobs. If necessary, change Queue in combo box.
correct range of events is set for event processing.
Threshold on rms is good for calibrated detector(s). Evaluation of this threshold requires in simple analysis which can be done after the 1st iteration on event averaging. Files containing averaged intensities and their rms values can be plotted as image and spectra. Spectrum of rms gives a rough idea of how to set this threshold. When threshold is set, second iteration on event averaging will produce correct mask-file for hot pixels, which rms exceeds the threshold.
How to calibrate / deploy files for dark correction
- Click on tab Dark.
- Set Instrument, Experiment, from combo boxes and click on Select button to select Detector(s) from pop-up window.
- Set filter for runs using combo boxes to the right from labels Show runs and for detectors.
- For runs in the table, which you want to calibrate, click on button Go, relax and wait until button Deploy show-up to the right from button. A few buttons Go can be clicked sequentially for different runs in order to run appropriate jobs in batch simultaneously.
- Change validity range if necessary, click on Deploy button. If everything is ok, it is done!
Known problems and ways around
World is not perfect. Anything may happen during this long workflow. Expert mode sheds light on most of the problems. Here is a list of most frequent problems;
- AFS token is expired for your terminal session. Batch system will complain. Before launching
calibman
application, first executekinit
andaklog
commands. - job on data from NEH/FEH are submitted to the wrong butch queue psfehq/psnehq. Set appropriate queue.
- DB has a list of sources (detectors) which does not coincide with what is written in xtc file. After job is submitted, check the list of detectors found in xtc scan and check only available detectors in the Detector selection menu.
Command line interface
Available since release ana-0.10.16.
Command line interface for dark-run calibration of detectors is implemented by P.O.C.s request. It can be started by the command calibrun
with parameters. The list of parameters can be dumped on terminal by the command:
Code Block |
---|
calibrun -h |
which prints
Code Block |
---|
Usage: calibrun [options] args
Example: calibrun -e xppi0613 -d CSPAD,OPAL1000 -c ./calib -P -D -r 173
calibrun - command line calibration utility
Options:
-h, --help show this help message and exit
-e EXP, --exp=EXP experiment name, ex.: cxi12345
-r RUNNUM, --run=RUNNUM
run number
-n NUM_EVENTS, --num_events=NUM_EVENTS
number of events to process
-s SKIP_EVENTS, --skip_events=SKIP_EVENTS
number of events to skip before processing
-m SCAN_EVENTS, --scan_events=SCAN_EVENTS
number of events to scan data and search for selected
detector(s)
-f EVENT_CODE, --event_code=EVENT_CODE
event code for dark event selection
-t THR_RMS, --thr_rms=THR_RMS
threshold on rms for hot pixel mask evaluation
-v RUNRANGE, --runrange=RUNRANGE
|
Code Block |
Usage: calibrun [options] args Example: calibrun -e xppi0613 -d CSPAD,OPAL1000 -c ./calib -P -D -r 173 calibrun - command line calibration utility Options: -h, --help show this help message and exit -e EXP, --exp=EXP validity experimentrun namerange, ex.: cxi12345123-567 or 123-end -rq RUNNUMQUEUE, --runqueue=RUNNUMQUEUE runqueue number name, ex: -n NUM_EVENTS, --num_events=NUM_EVENTS psnehq, psfehq, psanacsq. If specified number of- eventsrun toin processbatch -sw SKIP_EVENTSWORKDIR, --skip_events=SKIP_EVENTSworkdir=WORKDIR numberwork ofdirectory, events to skip before processingex: ./work -tc THR_RMSCALIBDIR, --thr_rms=THR_RMScalibdir=CALIBDIR non-standard calibration directory, ex: ./calib -x XTCDIR, --xtcdir=XTCDIR threshold on rms for hot pixel mask evaluation -v RUNRANGE, --runrange=RUNRANGE non-standard path to xtc directory(<INS>/<EXP>/xtc are validity run range, ex.: 123-567 or 123-end -q QUEUE, --queue=QUEUE required), ex: .mydir/xpp/xppi0613/xtc, or queue name, ex: psnehq, psfehq, psanacsq. If specified /reg/d/ffb/cxi/cxi12345/xtc -d DETECTOR, --detector=DETECTOR - run in batch detector names -w WORKDIR, --workdir=WORKDIRseparated by comma, work directory, ex: ./workCSPAD,CSPAD2x2,PNCCD -cp CALIBDIRPRINT_BITS, --calibdir=CALIBDIRprint_bits=PRINT_BITS 0-print nothing, non-standard calibration directory, ex: ./calib -d DETECTOR, --detector=DETECTOR +1-debug, +2-info, +4-warning, +8-error, +16-critical; default=036, minimal detector names separated by comma, recommended=020 -P, --process process ex:CSPAD,CSPAD2x2,PNCCD -p PRINT_BITS, --print_bits=PRINT_BITS xtc files and produce calib files under the =0-minimal print+warnings, =1-detailed printout "work" directory -PD, --processdeploy processdeploy xtccalibrated files andunder producethe "calib" files under thedirectory -L, --loadcfg load and use configuration parameters from file after "work" directory -D, --deploy deploy calibrated files under the "calibcalibman" directory |
The calibrun
application re-uses reuses code and approach developed for calibman
. At start calibrun
loads parameters from configuration file confpars-calibman.txt
, which by default . By default this file is the same configuration file as for calibman.
If necessary parameters are were already set in confpars-calibman.txt,
then calibrun can be started with a single parameter - run number (-r <run-number>) and options for processing (-P) and deployment (-D), for example:
Code Block |
---|
calibrun -r <run-number> -P -D |
otherwise, desired parameters need to be defined in the command line. Starting from the bare command calibrun
will print hint-messages If parameters are not defined either in the command line or in the configuration file, then
application will dump to terminal hint-warnings about missing option(s) which needs to be added to the command line in order to complete job. An example of the command with minimal set of self-sufficient parameters ,should looks like
Code Block |
---|
calibrun -e xppi0613 -d CSPAD,OPAL1000 -P -D -r 173 |
This command will process (-P) xtc files for experiment (-e) xppi0613, run (-r) 173, detectors (-d) CSPAD and (,) OPAL1000, and deploy (-D) them for default run range 173-end in under the default calib directory /reg/d/psdm/xpp/xppi0613/calib/
By default the image-averaging psana job is processed interactively. If queue (-q <queue-name>) option is provided, it is submitted in batch.
By default this the calibrun
command produces quite extensive output on terminal window, which in most cases is enough sufficient to figure out what is going on. If the print-control option (-p 0) is provided, the major output will be suppressed and only begin, end of job comments, and warning messages will be printed. Most complete output can be found in the session log file auto-saved in the work directory with relevant time stamp in the name, for example: ./work/2014-04-16-16:43:52-log.txt
Calibration manager calibman
, using the same file name conventions, can always be used to inspect all output files and problems generated by calibrun.
References
./work/2014-04-16-16:43:52-log.txt
GUI-based calibration manager calibman
has the same file name conventions and can always be used to inspect all output files and problems generated by the command line application calibrun.
How to run on ffb data
Command calibrun
can work on ffb
data using option -x <path-to-xtc>
. This option sets non-standard path to xtc files. For example:
Code Block |
---|
calibrun <options> -x /reg/d/ffb/cxi/cxi12345/xtc <more-options> |
The "live" mode is turning on automatically if the directory path has "ffb" sub-folder.