Motivation
Development of this application was stimulated by the discussion with Marcin Sikorski, doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function
g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),
where I(t)
is an image intensity at time t
, and tau
is a delay between two measurements.
Typical experimental condition can be described as follows:
- Run duration is about one hour at frequency up to 120 Hz that gives up to 10^5-10^6 images.
- Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
- Need to calculate
g2(tau)
for each pixel, averaged over all possible image timest
with time differencetau
between images. - A set of
tau
should have about 30-100 points in log scale uniformly covering the run duration. - Use for example xcsi0112-r0015: 500 images with 8 sec delay between images.
Desired time for evaluation of the auto-correlation function should be comparable with run duration <1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.
Algorithm
Basic idea is (1) to split image vs time for small parts in image, (2) to process each part on separate computer node, (3) to merge results at the end of processing. It is clear that significant speedup (about T/N_nodes_) is achieved at the 2nd stage. These three stages are performed in separate C++ applications. Wrapping python script allows to submit job by a single command. It takes care about file and sub-process management in this job, as described below.
Code location
All modules for this application resides in the package ImgAlgos:
Module |
Functionality |
---|---|
ImgVsTimeSplitInFiles |
splitter |
CorAna |
base class with common methods |
CorAnaData |
data processing for split files |
CorAnaInputParameters |
provides storage for input parameters |
CorAnaMergeFiles |
merging algorithm |
CorAnaPars.py |
singleton class for parameter storage in the wrapping file manager |
CorAnaSubmit.py |
global methods for the file manager |
app/corana_submit |
sequence of procedures in the file manager |
app/corana.cpp |
main module for the part of image vs time correlation processing |
app/corana_merge.cpp |
main module for merging |
data/psana-...-auto-correlation.cfg |
psana configuration file for ImgVsTimeSplitInFiles |
data/PlotCorAnaResults.py |
example of the python script which plots the test result histogram |
Image splitting
Image splitting is implemented as a regular psana module ImgAlgos::ImgVsTimeSplitInFiles.
Command to run interactively on psana####
or submit in batch from pslogin##
node:
psana -c <config-file> <xtc-file-list> bsub -q psfehq -o log-file 'psana -c <config-file> <xtc-file-list>'
For example:
psana -c ImgAlgos/data/psana-xcsi0112-r0015-img-auto-correlation.cfg /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-*
where ImgAlgos/data/psana-xcsi0112-r0015-img-auto-correlation.cfg
is an example of the configuration script for psana
and /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-*
are the input xtc files for particular run.
A couple of limitations due to LCLS policy:
Interactive job can be run on psana####
computer, but the batch queues are not seen from psana####
nodes...
Batch job can be submitted from pslogin##
computer, but data are not seen directly from pslogin##
nodes...
Produce files:
img-xcs-r0015-b0000.bin - file with a part of image vs time img-xcs-r0015-b0001.bin img-xcs-r0015-b0002.bin img-xcs-r0015-b0003.bin img-xcs-r0015-b0004.bin img-xcs-r0015-b0005.bin img-xcs-r0015-b0006.bin img-xcs-r0015-b0007.bin img-xcs-r0015-time.txt - list of time-records for all events in processed run. img-xcs-r0015-time-ind.txt - list of time-records for all events in processed run with time index. img-xcs-r0015-med.txt - file with metadata. In particular it has the original image size, number of image parts for splitting, number of images in run, etc.
Algorithms:
- The <int16_t> image data array is split for ordered (in psana *.cfg file) number of equal parts and each part is saved in the output
img-xcs-r0015-b####.bin
file sequentially for all selected events. - The appropriate time record for selected event is saved in the file
img-xcs-r0015-time.txt
. - At the end of the splitting procedure:
- the average time difference and its rms between sequential events is evaluated for all recorded time records.
- The file
img-xcs-r0015-time.txt
is re-processed and for each record the time index is evaluated as unsigned value of<time-index> = (<event-time> + 0.5 <average-time-between-events>) / <average-time-between-events>
- Event record with time index is saved in the file
img-xcs-r0015-time-ind.txt
- All metadata parameters which are required for further processing, such as input parameters, image size,
<average-time-between-events
, maximal value of the time index etc., are saved in fileimg-xcs-r0015-med.txt
.
Time correlation processing
ImgAlgos/app/corana
application
Command to run interactively on psana####
or submit in batch from pslogin##
node:
corana -f <fname-data> [-t <fname-tau>] [-h] [-l <logfile>] [-b <basedir>] bsub -q psfehq -o log-file 'corana -f <fname-data> [-t <fname-tau>] [-h] [-l <logfile>] [-b <basedir>]'
For example the interactive and batch mode commands:
corana -f img-xcs-r0015-b0001.bin -t my-tau.txt bsub -q psfehq -o log-file 'corana -f img-xcs-r0015-b0000.bin'
Where ~/<path-from-home-dir>/
is an arbitrary directory, but this directory should exist before running this script.
Produce files:
img-xcs-r0015-tau.txt - string of {{tau}} values for which the auto-correlation function is evaluated img-xcs-r0015-b0000-result.bin - auto-correlators for the part of the image for all {{tau}} values img-xcs-r0015-b0001-result.bin img-xcs-r0015-b0002-result.bin img-xcs-r0015-b0003-result.bin img-xcs-r0015-b0004-result.bin img-xcs-r0015-b0005-result.bin img-xcs-r0015-b0006-result.bin img-xcs-r0015-b0007-result.bin
Merging results
ImgAlgos/app/corana_merge
application
Command to run interactively on psana####
or submit in batch from pslogin##
node:
corana_merge -f <fname-data> [-t <fname-tau>] [-h] [-l <logfile>] [-b <basedir>] bsub -q psfehq -o log-file 'corana_merge -f <fname-data> [-t <fname-tau>] [-h] [-l <logfile>] [-b <basedir>]'
For example:
corana_merge -f img-xcs-r0015-b0001-result.bin -t my-tau.txt
This procedure produces file:
img-xcs-r0015-image-result.bin
Wrapping script
ImgAlgos/app/corana_submit
- this script allows to run all of above procedures by a single command from pslogin##
node and it keeps eye on processing of jobs in batch and doing the file managment. Command to start:
corana_submit [-c <config-file>] [-t <fname-tau>] [-x] <xtc-file-list>
For example:
corana_submit -c ImgAlgos/data/psana-xcsi0112-r0015-img-auto-correlation.cfg -t my-tau.txt /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc
File formats
- File with split-image data for selected events
img-xcs-r0015-b000N.bin
:
Currently this file contains<uint16_t>
amplitude for each pixel in binary format for:<data-for-img-partN-of-img1> <data-for-img-partN-of-img2> ... <data-for-img-partN-of-imgLast>
- File with metadata parameters
img-xcs-r0015-med.txt
:IMAGE_ROWS 1300 IMAGE_COLS 1340 IMAGE_SIZE 1742000 NUMBER_OF_FILES 8 BLOCK_SIZE 217750 REST_SIZE 0 NUMBER_OF_IMGS 500 FILE_TYPE bin DATA_TYPE uint16_t TIME_SEC_AVE 8.088413 TIME_SEC_RMS 0.063639 TIME_INDEX_MAX 499
- File with image time records
img-xcs-r0015-time.txt
:where each record has:1 0.000000 0.000000 20120616-080236.671607864 5366 0 2 8.026429 8.026429 20120616-080244.698036743 8255 1 3 16.144788 8.118359 20120616-080252.816395836 11177 2 4 24.154835 8.010048 20120616-080300.826443448 14060 3 ...
<image-in-file#> <t(sec)-from-the-1st-event> <dt(sec)> <time-stamp> <fiducials> <event#-since-configure>
- File with image time records and evaluated time index
img-xcs-r0015-time-ind.txt
:where each record has:1 0.000000 0.000000 20120616-080236.671607864 5366 0 0 2 8.026429 8.026429 20120616-080244.698036743 8255 1 1 3 16.144788 8.118359 20120616-080252.816395836 11177 2 2 4 24.154835 8.010048 20120616-080300.826443448 14060 3 3 5 32.281937 8.127102 20120616-080308.953545010 16985 4 4 ...
<image-in-file#> <t(sec)-from-the-1st-event> <dt(sec)> <time-stamp> <fiducials> <event#-since-configure> <time-index-starting-from-0>
- File with split-image correlators for each value of
tau
img-xcs-r0015-b000N-result.bin
:
Currently it saves<float>
correlator for each pixel in binary format for:<corr-for-img-partN-of-tau1> <corr-for-img-partN-of-tau2> ... <corr-for-img-partN-of-tauLast>
my-tau.txt
:contains the1 3 5 7 9 10 12 14 16 18 20 24 28 30 32 36 40 ... 160 180 200 240 280 300 320 360 400
tau
values presented in terms of number of ordered images in the file.
Quick start guide
We assume that everything is set up to work on LCLS analysis farm, otherwise see Computing and Account Setup.
How to run this procedure
If the version of the package ImgAlgos is already current software release, then you may run the script command(s) directly, for example:
cd <your-favorite-directory> mkdir work_corana sit_setup corana_submit [-c <config-file>] [-t <fname-tau>] [-x] <xtc-file-list>
If the code in package ImgAlgos has been recently changed and the updated release is not yet available, then one need to create the local release directory, get the latest/HEAD version of the package and compile the code as shown below:
cd <your-favorite-directory> newrel ana-current myReleaseDirectory cd myReleaseDirectory sit_setup addpkg ImgAlgos HEAD scons
Where to find results
The procedure will produce a bunch of files in the work_corana
directory. The most important files are
File name tail |
Format |
Content |
---|---|---|
*-image-result.bin |
binary for <float32_t> |
correlators for all image pixels for all tau values |
*-time-ind.txt |
text |
time records for all selected events/images |
*-tau.txt |
text |
the list of tau intervals |
*-med.txt |
text |
meta data parameters |
*-hist.txt |
text |
Histogram array with correlators averaged for ring regions of the image for all |