Motivation
Development of this application was stimulated by discussion with Marcin Sikorski, doing xcs experiments.
Experimentalists need in real-time algorithm for calculation of image vs time self-correlation function g2.
Typical experimental condition:
- Run duration is about one hour at frequency up to 120 Hz gives up to 500-10^6 images.
- Use Princeton camera with 1300x1340 pixels.
- Need to calculate g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>) for each pixel, averaged over all possible image times t with time difference tau between images.
- A set of tau should have about 30-100 points in log scale uniformly covering the run time duration.
- Use for example xcsi0112-r0015: 500 images with 8 sec between images.
Desired processing time should be comparable with run duration ~<1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.
Algorithm
Basic idea is (1) split image vs time for small parts in image, (2) process each part on separate computer node, (3) merge results at the end of processing. It is clear tignificant speedup (~T/Nnodes) is achieved at the 2nd stage. These three stages are performed in separate C++ applications. Wrapping python script allows to submit job by a single command. It takes care about numerous file and sub-process management in this job, as discussed below.
Code location
Image splitting
Image splitting is implemented as a regular psana module ImgAlgos::ImgVsTimeSplitInFiles.
Command:
psana -c <config-file> <xtc-file-list>
For example:
./CorAnaSubmit.py c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015*
Produce files:
img-xcs-r0015-b0000.bin img-xcs-r0015-b0001.bin img-xcs-r0015-b0002.bin img-xcs-r0015-b0003.bin img-xcs-r0015-b0004.bin img-xcs-r0015-b0005.bin img-xcs-r0015-b0006.bin img-xcs-r0015-b0007.bin img-xcs-r0015-time.txt img-xcs-r0015-med.txt
Time correlation processing
ImgAlgos/test/corana
application
Command:
<path>/corana -f <fname-data> -t <fname-tau> -h -l <logfile> -b <basedir>
For example:
<path>/corana -f img-xcs-r0015-b0001.bin -t my-tau.txt
Produce files:
img-xcs-r0015-tau.txt img-xcs-r0015-b0000-result.bin img-xcs-r0015-b0001-result.bin img-xcs-r0015-b0002-result.bin img-xcs-r0015-b0003-result.bin img-xcs-r0015-b0004-result.bin img-xcs-r0015-b0005-result.bin img-xcs-r0015-b0006-result.bin img-xcs-r0015-b0007-result.bin
Merging results
ImgAlgos/test/corana_merge
application
Command:
<path>/corana_merge -f <fname-data> -t <fname-tau> -h -l <logfile> -b <basedir>
For example:
<path>/corana_merge -f img-xcs-r0015-b0001-result.bin -t my-tau.txt
Produce file:
img-xcs-r0015-image-result.bin
Wrapping script
./CorAnaSubmit.py
./CorAnaSubmit.py -c <config-file> -t <fname-tau> -x <xtc-file-list>
For example:
./CorAnaSubmit.py -c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg -t my-tau.txt /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc
Where my-tau.txt
:
1 3 5 7 9 10 12 14 16 18 20 24 28 30 32 36 40 50 60 70 80 90 100 120 140 160 180 200 240 280 300 320 360 400
contains the tau values presented in terms of number of ordered images in the file.