Motivation
Development of this application was stimulated by the discussion with Marcin Sikorski, doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function
g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),
where I(t) is an intensity at time t, and tau is a delay between two measurements.
Typical experimental condition can be described as follows:
- Run duration is about one hour at frequency up to 120 Hz that gives up to 500-10^6 images.
- Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
- Need to calculate
g2(tau)
for each pixel, averaged over all possible image timest
with time differencetau
between images. - A set of
tau
should have about 30-100 points in log scale uniformly covering the run duration. - Use for example xcsi0112-r0015: 500 images with 8 sec between images.
Desired time for evaluation of the auto-correlation function should be comparable with run duration ~<1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.
Algorithm
Basic idea is (1) to split image vs time for small parts in image, (2) to process each part on separate computer node, (3) to merge results at the end of processing. It is clear that significant speedup (~T/Nnodes) is achieved at the 2nd stage. These three stages are performed in separate C++ applications. Wrapping python script allows to submit job by a single command. It takes care about numerous files and sub-process management in this job, as discussed below.
Code location
All modules for this application resides in the package ImgAlgos.
Image splitting
Image splitting is implemented as a regular psana module ImgAlgos::ImgVsTimeSplitInFiles.
Command:
psana -c <config-file> <xtc-file-list>
For example:
psana -c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-*
Produce files:
img-xcs-r0015-b0000.bin - file with a part of image vs time img-xcs-r0015-b0001.bin img-xcs-r0015-b0002.bin img-xcs-r0015-b0003.bin img-xcs-r0015-b0004.bin img-xcs-r0015-b0005.bin img-xcs-r0015-b0006.bin img-xcs-r0015-b0007.bin img-xcs-r0015-time.txt - list of time-record for all events in processed run. img-xcs-r0015-med.txt - file with metadata. In particular it has the original image size, number of image parts for splitting, number of images in run, etc.
File formats
img-xcs-r0015-b000N.bin
:
Currently it saves uint16_t amplitude for each pixel in binary format for:<data-partN-of-img1> <data-partN-of-img2> ... <data-partN-of-imgLast>
img-xcs-r0015-med.txt
:IMAGE_ROWS 1300 IMAGE_COLS 1340 IMAGE_SIZE 1742000 NUMBER_OF_FILES 8 BLOCK_SIZE 217750 REST_SIZE 0 NUMBER_OF_IMGS 500 FILE_TYPE bin DATA_TYPE uint16_t TIME_SEC_AVE 8.088413 TIME_SEC_RMS 0.063639
img-xcs-r0015-time.txt
:where each record has:1 1339858956.672 0.000 20120616-080236.671607864 005366 00000 2 1339858964.698 8.026 20120616-080244.698036743 008255 00001 3 1339858972.816 8.118 20120616-080252.816395836 011177 00002 4 1339858980.826 8.010 20120616-080300.826443448 014060 00003 ...
{{<image-in-file#> <dt(sec)> <time-stamp> <fiducials> <event#-since-configure> }}
Time correlation processing
ImgAlgos/test/corana
application
Command:
<path>/corana -f <fname-data> [-t <fname-tau>] -h [-l <logfile>] [-b <basedir>]
For example the interactive and batch mode commands:
<path>/corana -f img-xcs-r0015-b0001.bin -t my-tau.txt bsub -q psnehq -o ~/<path-from-home-dir>/log.txt '<path>/corana -f img-xcs-r0015-b0000.bin'
Where ~/<path-from-home-dir>/
or <path>
is arbitrary choice and is a matter of taste.
Produce files:
img-xcs-r0015-tau.txt - string of tau values for which the auto-correlation function is evaluated img-xcs-r0015-b0000-result.bin - auto-correlators for the part of the image for all tau values img-xcs-r0015-b0001-result.bin img-xcs-r0015-b0002-result.bin img-xcs-r0015-b0003-result.bin img-xcs-r0015-b0004-result.bin img-xcs-r0015-b0005-result.bin img-xcs-r0015-b0006-result.bin img-xcs-r0015-b0007-result.bin
Merging results
ImgAlgos/test/corana_merge
application
Command:
<path>/corana_merge -f <fname-data> [-t <fname-tau>] -h [-l <logfile>] [-b <basedir>]
For example:
<path>/corana_merge -f img-xcs-r0015-b0001-result.bin -t my-tau.txt
Produce file:
img-xcs-r0015-image-result.bin
Wrapping script
./CorAnaSubmit.py
./CorAnaSubmit.py -c <config-file> -t <fname-tau> [-x] <xtc-file-list>
For example:
./CorAnaSubmit.py -c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg -t my-tau.txt /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc
Where my-tau.txt
:
1 3 5 7 9 10 12 14 16 18 20 24 28 30 32 36 40 50 60 70 80 90 100 120 140 160 180 200 240 280 300 320 360 400
contains the tau values presented in terms of number of ordered images in the file.