You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

Features

Calibration Management Tool is intended to help in population of the calibration database. Most of imaging devices need in intensity corrections such as dark rate subtraction, per-pixel gain factor, masking of bad pixels etc. Some of imaging devices composed of separate sensors (a.k.a. tiles, segments, pixel matrices, etc.) such as CSPAD need in pixel geometry calibration parameters. There are many separate scripts based on python, psana framework, etc. which allow to get calibration files for different imaging devices. However, it is not easy to set correct parameters in order to run these scripts. One has to be careful in selection of input experiment name, run number, device name(s), to set correct chain of processing algorithms, and finally define the correct destination path for calibration files. Calibration Management Tool greatly simplify this task. It is designed to integrate access to many separate scripts under single Graphical User Interface (GUI), it knows about all possible variables and names, which can be selected from drop down menus, it helps to perform calibration and deploy calibration files in correct final directory with correct names.

Calibration Management Tool main functionality can be listed as follows:

  • helps to select and set instrument, experiment, and detector(s) which need to be calibrated,
  • easy switch between different tools such as Dark, Gain, Geometry calibration, ROI mask editor, File manager, etc. (not necessarily all of them are implemented),
  • issue necessary commands to OS or submit jobs in batch to perform calibration algorithms on data and get files with calibration parameters,
  • perform control and monitoring of the batch jobs,
  • perform monitoring of the calibration file quality,
  • deploy obtained or available calibration files in right sub-directories under /reg/d/psdm/<instrument>/<experiment>/calib/ with right names associated with a run range.

Users' Interface

User Interface (UI) of the Calibration Management Tool is implemented in Python using PyQt4, matplotlib, and other packages.  Main window of the UI shows up at start of calibman application. Its content may be different for the first and next sessions; current session status is preserved in configuration file at exit and restored at start of the next session (if this option is elected in configuration parameters). For example, UI with  Dark  tool is shown below.

Main window of UI consists of three panels from top to bottom:

  • Instrument/Experiment/Detector selection panel,
  • Tab-bar with switching application panel,
  • Log-book information panel.

Instrument/Experiment/Detector selection panel

This panel contains three selection buttons with preceded labels for

  • Instrument,
  • Experiment, and
  • Detector, 

which help to set associated parameters. Each of these buttons must be sequentially clicked and desired value should be selected from drop-down menu. After experiment name is selected, the default calibration directory appears in the next-right information field. This top-level calibration directory will be used as a destination path for deployment of all calibration files. Optional button Browser launch directory selection dialog window which allows to set different from default destination calibration directory. At first setting the sequence of operations in this panel is guided by the next enabled green button. Already enabled buttons can be re-selected at any time. For tools like Dark or Gain the list of calibrated detectors is selected by the check boxes in the pop-up panel

and the title of the Detector button shows how many detectors are selected. For File Manager tool only one detector should be selected.

Parameters of this panel must be set before any other actions with Calibration Manager.

 

 

Log-book panel

Log-book panel is located in the bottom of the main window. It is intended as a global information panel in order to minimize a number of specific information fields. Records and messages at variation of status or at clicks on information buttons are posted in this panel. At the end of each session content of this panel is saved under work directory in the log-file with name containing start session time stamp.

 

Status panel

Status panel shows up in a few applications such as Dark, File Manager, etc. on the top of application space. In most cases it shows content of the calibration directory for specified list of detectors. For each calibration file its effective run range is shown next right  to its name. The effective run range is presented by two numbers, begin run and end run of the range. Run ranges may overlap for different files. If this happens, files from bottom side have higher priority.

 

Tab-bar for tool selection

Tab-bar allows to switch between different applications. Completely implemented applications under tabs:

Dark

This tool is intended to produce calibration files for pedestals (dark level)  from dark runs and deploy them in correct place in calibration directory. In parallel hot pixel masks are produced and can be also deployed.

Dark panel is activated at click on Dark tab. It consists of

Control bar contains

      

  • buttons for list of runs filter by the run type and detector type and
  • check box for deployment of the hot pixel mask in parallel with averaged dark

List of run items panel consists of records for each run in "regular" - short or "expert" - extended mode.

      

Each run item in "regular" mode contains

      

  • button with triangle sign to switch between "regular" and "expert" mode of the run item,
  • run number
  • two fields to set run validity range for output calibration file
  • button Go - submits job in batch for evaluation of pedestals and hot pixel mask
  • button Deploy - shows up when butch job is completed successfully and all files for deployment are available.

Expert mode contains more information buttons for each run item

      

  • xtc Files - prints in the Log book panel the list of available xtc files,
  • Sources DB - prints in the Log book panel the list of available data sources found in DB,
  • Sources XTC - prints in the Log book panel the list of available data sources found at the xtc file scan,
  • O/Files - prints in the Log book panel the list of files associated with dark run processing and their current status,
  • Show cmd - prints in the Log book panel the list of deployment commands, and
  • More options check box - if it is marked then more buttons show up:
  • View - launch text-file viewer and
  • Plot - plots

 

 

ROI

Application panel under ROI (Region Of Interest) tab has a minimal number of control fields, which allow to specify the file with image and open/close Mask Editor UI in separate window.

Then the Mask Editor works as completely independent application. The Mask Editor allows to draw on image any number of a few geometry figure types and finally generate and save positive or negative mask for loaded image.

 

 

File Manager

 

 

Configuration

Configuration table has its own tab bar switching between panels with set of parameters.

Two panels are implemented currently

Configuration Parameters

 

 

Configuration File

Confirmation windows

There is a bunch of confirmation popup windows with modal-style dialog. Appropriate selection need to be done by clicking on possible continuation buttons.

 

Project status

Current status corresponds to:

relinfo
Pkg/Rel       TEST    BASE(ana-0.10.10)
CalibManager  HEAD   V00-00-10

Completely implemented applications under tabs:

  • Dark
  • ROI
  • File Manager
  • Configuration

How to get and run calibman

Run application in current release

If there is no known recent changes in the code of the project, then it may be launched using current release (≥ ana-0.10.10) from any directory :

cd <your-favorite-directory>;
sit_setup;
calibman;

 

Get latest version, compile, and run

The latest version of this application with recent modifications for packages can be created and executed from the local release directory. Login to psana, then run use commands with appropriate substitutions for <...> fields:

cd <your-favorite-test-directory>;
newrel ana-current test-CalibManager;
cd test-CalibManager;
sit_setup;
addpkg CalibManager HEAD;
addpkg <package2> HEAD;
addpkg <package3> HEAD;
...
scons;
calibman;

 

What to do next

  • Set correct queue which will be used for batch jobs: click on tabs: Configuration -> Parameters, select Queue combo box.
  • Click on tab Dark.
  • Set Instrument, Experiment, from combo boxes and click on Select button to select Detector(s) from pop-up window.
  • Set filter for runs using combo boxes to the right from labels Show runs and for detectors
  • For runs in the table, which you want to calibrate, click on button Go, relax and wait until button Deploy show-up to the right from button 
  • Change validity range if necessary, click on Deploy button.  If everything is ok, it is done!

 

 

Known problems and ways around

World is not perfect. Anything may happen during this long workflow. Expert mode sheds light on most of the problems. Here is a list of most frequent problems;

  • AFS token is expired for your terminal session. Batch system will complain. Use aklog command.
  • work directory is missing... make it manually in accordance with selected or default path in Configuration -> Parameters, Dir work
  • job on data from NEH/FEH are submitted to the wrong butch queue psfehq/psnehq. Set appropriate queue.
  • DB has a list of sources (detectors) which does not coincide with what is written in xtc file. After job is submitted, check the list of detectors found in xtc scan and check only available detectors.

References

  • No labels