Motivation
Development of this application was stimulated by the discussion with Marcin Sikorski (meeting on 2012-08-30), doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function
g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),
where I(t)
is an image intensity at time t
, and tau
is a delay between two measurements.
Typical experimental condition can be described as follows:
- Run duration is about one hour at frequency up to 120 Hz that gives up to 10^5-10^6 images.
- Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
- Need to calculate
g2(tau)
for each pixel, averaged over all possible image timest
with time differencetau
between images. - A set of
tau
should have about 30-100 points in log scale uniformly covering the run duration. - Use for example xcsi0112-r0015: 500 images with 8 sec delay between images.
Desired time for evaluation of the auto-correlation function should be comparable with run duration <1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.
In first approximation this problem was solved, as it is described in the [PCDS:Command Line Interface For Time Correlation Analysis]. However, in order to be useful, the command line algorithm needs to be integrated in the global analysis system, which de bene esse called as Integrated Analysis Environment for Time Correlation Experiments and is discussed below.
Problems and tentative solutions
Based on 2012-10-03 meeting:
In order to be useful this application should do correct math, accounts for image mask, discards bad events (noisy and "bright" pixels), apply normalization etc., and have a convenient GUI. Below is a list of requirements, marked as , with suggested solutions, marked as
if exists or as
if needs to be implemented.
Pedestals
"dark" run number should be provided by user and the imaging camera pedestals should be evaluated and applied for all runs until the "dark" run number has not changed.
For pedestals evaluation: use available
ImgAlgos::ImgAverage
psana module for "dark" run, which produces file with averaged over events pedestals and the file with their rms values.
For pedestals subtraction: use
ImgAlgos::ImgCalib
psana module in the same job which evaluates correlators; the pedestals will be subtracted and corrected image will be retained in the event and used for correlator calculations.
Low level threshold
Image pixel intensity physically can't be negative. Low amplitude noise should be suppressed by the threshold. The threshold amplitude should be provided by user (along with substituting amplitude).
Is implemented in the
ImgAlgos::ImgCalib
psana module, right after pedestal subtractuion.
Image filtering
Usually users use different type of intensity monitor signals in order to retain/discard image for/from further processing. Discarded images should not contribute into the correlation analysis. The spectra of intensity monitors should be available for browsing. User should be able to select the intensity monitor(s) from the list and set low and high thresholds.
The filtering module may be implemented in psana. Based on selected intensity monitor(s) and thresholds it will decide to retain or discard event and accumulate spectral histograms. The histograms will be saved in file at the end of run.
Control GUI should be able to browse the intensity monitor histograms and set the thresholds.
Selection of intensity monitors
It would be nice to have an algorithm like in XTC explorer
Possible options:
- run application as a plug-in for XTC Explorer,
- pyana module performing similar to XTC Explorer algorithm,
- stand-alone C++ module reading XTC datagrams,
- hardwired list of intensity monitors.
- psana module, which needs to be developed
Dynamic mask
Imaging camera may have permanently hot pixels or some pixels may be saturated during the run. User need to set a thresholds on hot pixels and high intensity.
If the pixel amplitude crosses the high intensity threshold at least once during the run, then this pixel should be excluded from further analysis. The same is valid for hot pixels, which shows above threshold intensity in large fraction of events.
This is implemented in the psana module
ImgAlgos::ImgMaskEvaluation
, which works before event selection algorithm. Two files of image size may be produced 1) for saturated and 2) for hot pixels.
Static mask
The beam-stopper region and some areas with fringes should be masked. It could be useful to have a graphical editor for mask.
See section for GUI.
Graphical editor for selected regions
Sometimes it is useful to select good region of the image. It could be convenient to use a graphical editor, as for mask.
See section for GUI.
Center of the image
User should have an option to set a center of the rings for histograms.
See section for GUI.
Correct normalization of g2
Evaluation of
g2
for image regions is not that simple as presented by the formula for a single pixel:
g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),
In order to get physically meaningful results for g2, the correlators <I(t)> and <I(t+tau)> should be averaged in the fine rings around center with number of bins N2, which is order of 100, with dR down to 1-2 pixels.
Then the <I(t)*I(t+tau)> correlator should be averaged over bold rings intended for G2 evaluation. The number of these rings N1 should be order of 10.
The N2 and N1 should be defined by user.
It might be useful to define the histogram region by the sector in the user-defined angular range.
In order to have required normalization of correlators, it is not enough to save the
g2
value only. So, the format of the resulting file has changed. Now for each value of tau
the output file contains the <I(t)>, <I(t+tau)>, and <I(t)*I(t+tau)>, each for entire image written in binary for float format. Not all masks, selection regions, normalization etc. are available during correlators calculation, so correlators are evaluated for all pixels. Which pixels should be included in the G2 for each region can be decided at the final stage of processing. This approach allows to perform the most time consuming procedure - the correlators calculation once and do the analysis after that.
See section for GUI.
GUI
In order to get an easy interface to all sub-processes, it seems to be useful to have a GUI with configuration of everything through the GUI.
Well, presumably users will want different specific features in their analyses which can not be foreseen in implementation of GUI. It is pretty unlikely that everything in analysis can be done clicking on buttons in GUI. Then, it could be nice if user understand what he is doing step by step and have a monitoring at the end of each stage. We are doing science, not a standard pre-defined things... Most generic way to process data is to have a separate procedures with command line interface.
Anyway, the browser/presenter of data stored in the files after pre-processing could be provided for a set of common plots.
All features listed in previous sections, such as static and dynamic mask, restriction of the region(s) of interest, selection of the image center, the binning scheme etc., can be done in the browser at the final stage of the analysis.
Generic GUI as a shell for entire analysis can be implemented as an interface to the command line procedures:
- Each command line procedure may have a dedicated GUI for procedure configuration and monitoring.
- All procedures may be listed in the main GUI with a status sign.
- The main issue of this approach is a cross-configuration between separate procedures. This can be achieved if all procedures will have a common list of configuration parameters.
Here is my comment
GUI Implementation
Updated on 2012-12-02
There are three layouts implemented for this application:
GUIMain
Layout 1:
- Many separate windows
- Plots are isolated
GUIMainTB
Layout 2:
- All input windows are integrated in one with tool bar
- Logger and File browser windows are isolated
- Plots are isolated
GUIMainSplit
Layout 3:
- All input windows are integrated in one with tool bar
- Logger window takes residual part in the bottom
- File browser window is isolated
- Plots are isolated
Code location
Everything will reside in CorAna
package.
Modules
Updated on 2012-12-10
All module names beginning with letters GUI
implements different Graphical User Interfaces.
Module |
Description |
---|---|
AppDataPath.py |
Local version with added path to data from src directory. |
BatchJobData.py |
|
BatchJobPedestals.py |
Class contains methods for batch job submission and monitoring for pedestals. |
BatchLogParser.py |
|
ConfigFileGenerator.py |
Class has methods to generate psana configuration and other scripts from stubs located in |
ConfigParameters.py |
Base class for configuration parameters. |
ConfigParametersCorAna.py |
Sub-class for |
CorAnaUtils.py |
Junk for eamples. |
FileNameManager.py |
Class dynamically generates all file names for current configuration parameters. |
GUIAnaPartitions.py |
|
GUIAnaSettings.py |
|
GUIAnaSettingsLeft.py |
|
GUIAnaSettingsOptions.py |
|
GUIAnaSettingsRight.py |
|
GUIBeamZeroPars.py |
|
GUIBlamish.py |
Sub-GUI of GUIFiles.py |
GUICCDSettings.py |
|
GUIConfigParameters.py |
GUI for the configuration parameters file management. |
GUIDark.py |
Sub-GUI of GUIFiles.py - manipulations with dark runs |
GUIData.py |
|
GUIFileBrowser.py |
Text file browser for this project |
GUIFiles.py |
Central GUI for file settings contains tab-bar for other widget selection |
GUIFlatField.py |
Sub-GUI of GUIFiles.py |
GUIHelp.py |
WIdget for messages, i.e. help |
GUIImgSizePosition.py |
|
GUIInstrExpRun.py |
depricated GUI |
GUIIntensityMonitors.py |
Control GUI for intensity monitors |
GUIKineticMode.py |
|
GUILoadResults.py |
|
GUILogger.py |
GUI for logger |
GUIMain.py |
The first GUI at start of this application |
GUIMainSplit.py |
The same as GUIMainTB.py with integrated GUILogger.py |
GUIMainTB.py |
The first GUI at start of this application with tab bar |
GUINonKineticMode.py |
|
GUIRun.py |
|
GUISetupEnergyAngle.py |
|
GUISetupInfo.py |
|
GUISetupInfoLeft.py |
|
GUISetupInfoRight.py |
|
GUISetupPars.py |
|
GUISpecularPars.py |
|
GUISystemSettings.py |
|
GUISystemSettingsLeft.py |
|
GUISystemSettingsRight.py |
|
GUITransmissionPars.py |
|
GUIViewResults.py |
|
GUIWorkResDirs.py |
|
GlobalUtils.py |
Module contains all possible global methods. |
ImgSpeNavToolBar.py |
Re-implemented standard tool-bar (depricated) |
Logger.py |
Core class for logger |
Overlay.py |
Demonstration of how to draw something on the top of GUI |
PlotArray.py |
Main class for array presentation plot |
PlotArrayButtons.py |
Widget for custom button-bar |
PlotArrayWidget.py |
Widget for graphical window |
PlotImgSpe.py |
Main class for interactive plot with custom button bar |
PlotImgSpeButtons.py |
Widget for custom button-bar |
PlotImgSpeWidget.py |
Graphical image for image and spectral histogram |
PlotTime.py |
Main class for time record presentation plot |
PlotTimeWidget.py |
Widget for graphical window |
Status of the project
Status of the project on 2012-11-07
The system of GUIs, consisting of dozens of
CorAna.GUI...
modules is implemented in the draft approximation. Roughly it reproduces all features of the old program.
Infrastructural modules
ConfigParameters.py
, ConfigParametersCorAna.py
, and GUIConfigParameters.py
provides convenient approach for maintenance of all configuration parameters.
Infrastructural modules
Logger.py
and GUILogger.py
provides a generic approach to logging system.
Module
ConfigFileGenerator.py
use current settings of configuration parameters and stub-file scripts from CorAna/data/scripts/
and generates the psana configuration files.
Module
FileNameManager.py
is a single place which provides a dynamic file names for current version of the configuration parameters.
Module
BatchJobPedestals.py
is intended to prepare, submit, and monitor the butch job for pedestals evaluation.
BatchJobDataPreliminary.py
- get the list of PV variables, available intensity moniotors, photon beam intensity etc.
BatchJobFlatField.py
- to get the flat field averaged image if relevant run is available.
BatchJobCorAna.py
- the batch job for main calculation algorithm.
Graphical presentation for masks, pedestals, averaged image, partitions etc.
Graphical editor for the regions of interest,
Graphical presentation and GUI for final results.
2012-11-05 – 2012-11-11 Week progress
A src/PsanaConfigFileGenerator.py
A src/FileNameManager.py
A src/BatchJobPedestals.py
A src/AppDataPath.py
A src/ConfigFileGenerator.py
A src/GUIDark.py
A src/ImgSpeButtons.py
A src/ImgSpeNavToolBar.py
A src/ImgSpeWidget.py
A src/ImgSpeWithGUI.py
A data/scripts/psana-pedestals.cfg
A data/scripts/psana-tahometer.cfg
2012-11-12 – 2012-11-18 Week progress
A src/GUIFileBrowser.py - file browser for text files of this project.
A src/GUIHelp.py - pop-up box with message as a widget.
A src/GUIWorkResDirs.py - separate widget for setting of work and result output directory and common prefix file name.
In src/GUILogger.py and src/Logger.py - now messages are accumulated in 2d tuple. Add filtration by the message type "debug", "info", "warning", etc.
src/GUIFiles.py is significantly re-organized; now it is GUI with tabs.
All significant fields are moved in GUIs like src/GUIDark.py (GUIData, GUIFlat).
src/GUIDark.py - is re-organized.
In the interactive plot:
- In src/ImgSpeWithGUI.py the src/ImgSpeNavToolBar.py - is depricated
- src/ImgSpeButtons.py - re-organized: add save, reset, and menu buttons, log check-box.
- In src/ImgSpeWidget.py - the log functionality needs to be implemented...
2012-11-19 – 2012-11-25 Week progress
Suggestions from Marcin:
- Time(sec) - should be the averaged time between frame time-stamps.
- Spectral plot: log-scale for X-axis only.
- Flat field and Blamish files will be provided from external apps. No need to evaluate them now. Needs to be selected by the path.
- Substitute the start and end events after the scan.
Progress:
- Restracturization:
tahometer
replaced bypeds_scan
. - update the start and end fields after scan.
- remove psana log files before job submission in batch.
- the start and end fields are set from cofig.pars, if the scan log-file is availabe, they are updated from this file.
- reset fields to default if the input xtc file is changed.
- A src/GUIFlatField.py - file selection, plot, browse file
- A src/GUIBlamish.py
- A src/GUIMainTB.py - try different view for composite window
- A src/GUIMainSplit.py - try different view for composite window
- Add psana module: ImgAlgos::ImgTimeStampList - creates the time-stamp list and evaluates the mean time between frames and its RMS.
- in GUIDark.py use variables produced by the ImgAlgos::ImgTimeStampList.
2012-11-26 – 2012-12-02 Week progress
Suggestions from Marcin:
- Marcin prefers layout with separated GUILogger window, like in GUIMainTB.py
- Need in a tool reporting problems in time-stamp sequence.
- In GUIDark.py - untill the total number of events is unknown, use end "-1" and generate a warning if it is submitted in batch for pedestals.
- Add check-box showing if the Blamish and Flat-field files will be applied
- Marcin returned the paper-copy of GUIs with comments about titles and parameters.
- The list of variables for intensity monitor:
- IPM2 [CH0,...,CH3]
- IPMMON0 [...]
- IPM4 [...]
- IPM5 [...]
- GASDET [...]
- Needs in separate GUI for intensity monitors for normalization and veto.
Progress:
- In GUIBlamish.py and src/GUIFlatField.py add check-boxes.
- Apply Marcin's comments to GUIs with System, Setup etc. parameters, GUICCDCorrectionSettings.py is removed..
- In src/GUIMainTB.py optimized the GUI sizes.
- Changed names from GUIBatch... to GUISystem...
- New psana module [PCDS:Psana Module Catalog#Module ImgAlgos::IntensityMonitorsData], which gets data for equivalent HDF5 variables:
in psana they can be accessed as:
/Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataFEEGasDetEnergy/FEEGasDetEnergy /Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataIpimbV1/XCS-IPM-02 /Configure:0000/Run:0000/CalibCycle:0000/Bld::BldDataIpimbV1/XCS-IPM-mono /Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/XcsBeamline.1:Ipimb.4 /Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV2/XcsBeamline.1:Ipimb.5
and share pointers:m_srcFEEGasDetE = configStr("feeSource", "BldInfo(FEEGasDetEnergy)"); m_srcIPM2 = configStr("ipm2", "BldInfo(XCS-IPM-02)"); m_srcIPMMono = configStr("ipmmono", "BldInfo(XCS-IPM-mono)"); m_srcIPM4 = configStr("ipm4", "DetInfo(XcsBeamline.1:Ipimb.4)"); m_srcIPM5 = configStr("ipm5", "DetInfo(XcsBeamline.1:Ipimb.5)");
shared_ptr<Psana::Bld::BldDataFEEGasDetEnergy> fee = evt.get(m_srcFEEGasDetE); shared_ptr<Psana::Ipimb::DataV2> data2 = evt.get(m_src); // for m_srcIPM4 and m_srcIPM5 shared_ptr<Psana::Bld::BldDataIpimbV1> ipimb1 = evt.get(m_src_ipimb); // for m_srcIPM2 and m_srcIPMMono
- In GUISystemPars: add total number of events and dt+-rms to GUISystemPars.
- In GUIDark: add check-box and file-browser button with complete functionality.
- In GUIDark: add protection for pedestal job submission in batch with default number of events.
- In BatchJobPedestals: add protection against duplicated job submission in batch using timeout (now 100 sec) or until the previous job is DONE.
- In ImgSpeWidget: implemented the log-scale plot with full functionality and protections...
2012-12-03 – 2012-12-09 Week progress
No meeting this week due to experiments in xcs hatch.
Progress:
- In GUIFileBrowser: add one more parameters after the list of files, the file to start browser
- GUIData is implemented with full functionality including
- check box to activate/protect buttons
- "File" button and info field - selects the xtc file
- "Scan" button - submits scanner job in batch which
- counts events,
- saves in file the time records ind evaluate an average interval between frames
- saves in file the intensity monitor data
- "Average" button - submits job in batch for averaging image for the specified range of events
- "Check status" button - checks the status of both batch jobs, check if the file with averaged image is available, update the info fields for number of events and averaged time between frames
- "Check files" button - prints the list of work files for data and check their availability
- "Browse" button - open the file browser with a list of work files for data
- "img-Plot" button - plots the averaged image from file
- "t-Plot" button - shows the plot for time records saved in file
- "Remove" button - removes all work files from the work directory
- GUIIntensityMonitors: GUI for Intensity monitors
- PlotArray.py, PlotArrayButtons.py, PlotArrayWidget.py - plot for array presentation
- PlotTime.py, PlotTimeWidget.py - plot for time records presentation
2012-12-10 – 2012-12-16 Week progress
Q&A Marcin:
- Q: XcsBeamline.2 - is it used simultaneously with XcsBeamline.1 ?
- A: XcsBeamline.1
- Q: What control parameters should be added for intensity monitors?
- A: should be added:
- radio button for normalization or no-normalization
- filter data check box
- low-upper amplitude range
- A: should be added:
- Q: Layout of GUIMainTB.py with variable height, is it ok?
- A: Looks fine.
- Q: Layout variation for check-box - disappearance of buttons, is it ok?
- A: Change color of button-titles.
- Q: What image buttons are used as a mask editor etc.
- A: GUIAnalysisSettings:
- "MaskPolygon" button - for polygon editor on data image
- "Show Mask & Partitions" button - shows the mask and partitions on data image
- A: GUIAnalysisSettings:
Suggestions from Marcin:
- Add button to all plots and gui to submit them in ELog.
Change log presentation for image histogram - log for vertical scale only.
Other things to do
- Use intensity monitor names in generation of the psana.cfg file
- Check move window coordinates for plots... Some of them opens in unexpected places.
GUIDark and GUIData - inappropriate change of style for "end" field
Progress
- In GUIDark and GUIData - fixed issue with changing style for "end" field
- PlotImgSpeWidget.py - change log scale from x to y
- GUIIntensityMonitors.py - add more fields:
- radio buttons for normalization
- check boxes for selection
- edit fields for Imin and Imax values with changing behavior and style depending on radio and check box status. Check them for equity to default and Imin >= Imax