Motivation

Development of this application was stimulated by the discussion with Marcin Sikorski (meeting on 2012-08-30), doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function

g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),

where I(t) is an image intensity at time t, and tau is a delay between two measurements.
Typical experimental condition can be described as follows:

• Run duration is about one hour at frequency up to 120 Hz that gives up to 10^5-10^6 images.
• Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
• Need to calculate g2(tau) for each pixel, averaged over all possible image times t with time difference tau between images.
• A set of tau should have about 30-100 points in log scale uniformly covering the run duration.
• Use for example xcsi0112-r0015: 500 images with 8 sec delay between images.
  Desired time for evaluation of the auto-correlation function should be comparable with run duration <1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.

In first approximation this problem was solved, as it is described in the Command Line Interface For Time Correlation Analysis. However, in order to be useful, the command line algorithm needs to be integrated in the global analysis system, which de bene esse called as Integrated Analysis Environment for Time Correlation Experiments and is discussed below.

Problems and tentative solutions

Based on 2012-10-03 meeting:

In order to be useful this application should do correct math, accounts for image mask, discards bad events (noisy and "bright" pixels), apply normalization etc., and have a convenient GUI. Below is a list of requirements, marked as , with suggested solutions, marked as if exists or as if needs to be implemented.

Pedestals

"dark" run number should be provided by user and the imaging camera pedestals should be evaluated and applied for all runs until the "dark" run number has not changed.
For pedestals evaluation: use available ImgAlgos::ImgAverage psana module for "dark" run, which produces file with averaged over events pedestals and the file with their rms values.
For pedestals subtraction: use ImgAlgos::ImgCalib psana module in the same job which evaluates correlators; the pedestals will be subtracted and corrected image will be retained in the event and used for correlator calculations.

Low level threshold

Image pixel intensity physically can't be negative. Low amplitude noise should be suppressed by the threshold. The threshold amplitude should be provided by user (along with substituting amplitude).
Is implemented in the ImgAlgos::ImgCalib psana module, right after pedestal subtractuion.

Image filtering

Usually users use different type of intensity monitor signals in order to retain/discard image for/from further processing. Discarded images should not contribute into the correlation analysis. The spectra of intensity monitors should be available for browsing. User should be able to select the intensity monitor(s) from the list and set low and high thresholds.
The filtering module may be implemented in psana. Based on selected intensity monitor(s) and thresholds it will decide to retain or discard event and accumulate spectral histograms. The histograms will be saved in file at the end of run.
Control GUI should be able to browse the intensity monitor histograms and set the thresholds.

Selection of intensity monitors

It would be nice to have an algorithm like in XTC explorer
Possible options:

• run application as a plug-in for XTC Explorer,
• pyana module performing similar to XTC Explorer algorithm,
• stand-alone C++ module reading XTC datagrams,
• hardwired list of intensity monitors.
• psana module, which needs to be developed

Dynamic mask

Imaging camera may have permanently hot pixels or some pixels may be saturated during the run. User need to set a thresholds on hot pixels and high intensity.
If the pixel amplitude crosses the high intensity threshold at least once during the run, then this pixel should be excluded from further analysis. The same is valid for hot pixels, which shows above threshold intensity in large fraction of events.
This is implemented in the psana module ImgAlgos::ImgMaskEvaluation, which works before event selection algorithm. Two files of image size may be produced 1) for saturated and 2) for hot pixels.

Static mask

The beam-stopper region and some areas with fringes should be masked. It could be useful to have a graphical editor for mask.
See section for GUI.

Graphical editor for selected regions

Sometimes it is useful to select good region of the image. It could be convenient to use a graphical editor, as for mask.
See section for GUI.

Center of the image

User should have an option to set a center of the rings for histograms.
See section for GUI.

Correct normalization of g2

Evaluation of g2 for image regions is not that simple as presented by the formula for a single pixel:

g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),

In order to get physically meaningful results for g2, the correlators <I(t)> and <I(t+tau)> should be averaged in the fine rings around center with number of bins N2, which is order of 100, with dR down to 1-2 pixels.
Then the <I(t)*I(t+tau)> correlator should be averaged over bold rings intended for G2 evaluation. The number of these rings N1 should be order of 10.
The N2 and N1 should be defined by user.
It might be useful to define the histogram region by the sector in the user-defined angular range.
In order to have required normalization of correlators, it is not enough to save the g2 value only. So, the format of the resulting file has changed. Now for each value of tau the output file contains the <I(t)>, <I(t+tau)>, and <I(t)*I(t+tau)>, each for entire image written in binary for float format. Not all masks, selection regions, normalization etc. are available during correlators calculation, so correlators are evaluated for all pixels. Which pixels should be included in the G2 for each region can be decided at the final stage of processing. This approach allows to perform the most time consuming procedure - the correlators calculation once and do the analysis after that.

See section for GUI.

GUI

In order to get an easy interface to all sub-processes, it seems to be useful to have a GUI with configuration of everything through the GUI.
Well, presumably users will want different specific features in their analyses which can not be foreseen in implementation of GUI. It is pretty unlikely that everything in analysis can be done clicking on buttons in GUI. Then, it could be nice if user understand what he is doing step by step and have a monitoring at the end of each stage. We are doing science, not a standard pre-defined things... Most generic way to process data is to have a separate procedures with command line interface.
Anyway, the browser/presenter of data stored in the files after pre-processing could be provided for a set of common plots.
All features listed in previous sections, such as static and dynamic mask, restriction of the region(s) of interest, selection of the image center, the binning scheme etc., can be done in the browser at the final stage of the analysis.
Generic GUI as a shell for entire analysis can be implemented as an interface to the command line procedures:

• Each command line procedure may have a dedicated GUI for procedure configuration and monitoring.
• All procedures may be listed in the main GUI with a status sign.
• The main issue of this approach is a cross-configuration between separate procedures. This can be achieved if all procedures will have a common list of configuration parameters.

GUI Implementation

Updated on 2012-12-02

Three versions of GUI were implemented for this application. It was decided to use the GUIMainTB
layout, where:

• All input windows are integrated in one with tool bar
• Logger and File browser windows are isolated
• Plots are isolated

Code location

Everything resides in CorAna package, which is included in LCLS offline releases beginning from ana-0.7.16.

Modules

Updated on 2012-12-10
Notations:
- is done (at least it is assumed for now...)
- needs more work
- test or deprecated modules
All module names beginning with letters GUI implements different Graphical User Interfaces.

Status of the project

Updated on 2013-02-07

PSANA modules

For dark run pre-processing

Scaner
Runs in batch for the dark file to get preliminary information.
ImgAlgos.ImgTimeStampList

• counts number of events in the file
• makes file with time stamps
• evaluate time intervals between frames (for dark run)

Pedestals
Runs in batch for the dark file to get averaged pedestals.
ImgAlgos.PrincetonImageProducer gets image from event as an ndarray object
ImgAlgos.Tahometer evaluates performance of the batch job
ImgAlgos.ImgAverage produces file with averaged image for given range of events.

For data pre-processing

Scaner
Runs in batch for the data file to get preliminary information.
ImgAlgos.Tahometer evaluates performance of the batch job, counts number of events in the data file
ImgAlgos.ImgTimeStampList makes file with time stamps and time record counters for tau
ImgAlgos.IntensityMonitorsData makes file with intensity monitor records

Average
Runs in batch for the data file to get averaged image
ImgAlgos.Tahometer evaluates performance of the batch job
ImgAlgos.PrincetonImageProducer gets image from event as an ndarray object
ImgAlgos.ImgAverage produces file with averaged image for given range of events
ImgAlgos::ImgMaskEvaluation - module is available, but needs to be included in psana.cfg file for evaluation of:

• saturated mask
• noise mask
• combined mask

For data processing

Split
ImgAlgos.Tahometer
ImgAlgos.PrincetonImageProducer
ImgAlgos.ImgVsTimeSplitInFiles

Process
Data processing is implemented in stand alone (non-psana) modules
ImgAlgos.CorAna.cpp
ImgAlgos.CorAnaInputParameters.cpp
ImgAlgos.CorAnaData.cpp

Merge
CorAnaMergeFiles.cpp - saves binary file for float(32) with shape (Ntau,3,rows,cols), where 3 stands for <Ip>, <If>, and <Ip*If>
Can be acessed in python as

    sp.cor_arr = np.fromfile(sp.fname, dtype=np.float32)
    <image-size> = rows * cols
    nptau = <file-size>/<image-size>/3
    sp.cor_arr.shape = (nptau, 3, rows, cols)

GUI

The system of GUIs, consisting of dozens of CorAna.GUI... modules is implemented in the draft approximation. Roughly it reproduces all features of the old program.

Input parameters GUIs

CorAna.GUI... most of them are available. Will be added or extended if necessary.
Files - define input files and do pre-processing
Setup Info
Analysis Info
System
Intensity Monitor needs to be used at processing stage

Run GUI

Input - short summary of input info for data processing
Split - control and monitoring for the 1st stage of processing
Process - ... 2nd stage ...
Merge - ... 3d stage ...
Auto - ... for all 3 stages ...

View Results GUI

Contains a set of control fields for presentation of results

Graphics

PlotArray*.py - for Intensity Monitors
PlotImgSpe*.py - for images, partition maps, masks etc.
PlotTime*.py - for time stamp monitoring
PlotG2*.py - G2 plot peresentation
Mask editors...

Infrastructural modules

Infrastructural modules provide basic infrastructure of the project.
ConfigParameters.py, ConfigParametersCorAna.py, and GUIConfigParameters.py provides convenient approach for maintenance of all configuration parameters.
Infrastructural modules Logger.py and GUILogger.py provides a generic approach to logging system.
Module ConfigFileGenerator.py use current settings of configuration parameters and stub-file scripts from CorAna/data/scripts/ and generates the psana configuration files.
Module FileNameManager.py is a single place which provides a dynamic file names for current version of the configuration parameters.
GlobalUtils.py - global utilities for common operations
BatchJob.py - superclass for batch job submission
BatchJobPedestals.py - pre-processing for dark run files
BatchJobData.py - re-processing for data files
BatchJobCorAna.py - main data processing - calculation algorithm.

Progress

See Weekly progress of the IDPE project for TCE

To-do list

Masks
Evaluation at data pre-processing the noisy and saturated pixel mask
Graphical editor for regions of interest
Graphical editor for blemish mask

Processing (at psana Split level?)
Intensity monitors: needs to be used at processing in psana
Account for thresholds

View Results
geometry for reflected beam
proper combination for total mask in ViewResults.py
fits

ELog
Add submission using responce ID, when Igor provide an interface
GlobalUtils: comment/uncomment submission in ELog