Motivation

Development of this application was stimulated by the discussion with Marcin Sikorski, doing xcs experiments.
Users need in real-time algorithm for calculation of image vs time auto-correlation function

g2(tau) = <I(t)*I(t+tau)> / (<I(t)> * <I(t+tau)>),

where I(t) is an intensity at time t, and tau is a delay between two measurements.
Typical experimental condition can be described as follows:

• Run duration is about one hour at frequency up to 120 Hz that gives up to 500-10^6 images.
• Currently typical imaging devise is a Princeton camera with 1300x1340 pixels.
• Need to calculate g2(tau) for each pixel, averaged over all possible image times t with time difference tau between images.
• A set of tau should have about 30-100 points in log scale uniformly covering the run duration.
• Use for example xcsi0112-r0015: 500 images with 8 sec between images.
  Desired time for evaluation of the auto-correlation function should be comparable with run duration ~<1 hour. Currently this algorithm takes a few hours that can not be used for fast feedback in real time experiment.

Algorithm

Basic idea is (1) to split image vs time for small parts in image, (2) to process each part on separate computer node, (3) to merge results at the end of processing. It is clear that significant speedup (~T/Nnodes) is achieved at the 2nd stage. These three stages are performed in separate C++ applications. Wrapping python script allows to submit job by a single command. It takes care about numerous files and sub-process management in this job, as discussed below.

Code location

All modules for this application resides in the package ImgAlgos:

Image splitting

Image splitting is implemented as a regular psana module ImgAlgos::ImgVsTimeSplitInFiles.

Command:

psana -c <config-file> <xtc-file-list>

For example:

psana -c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg  /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-*

Produce files:

img-xcs-r0015-b0000.bin - file with a part of image vs time
img-xcs-r0015-b0001.bin
img-xcs-r0015-b0002.bin
img-xcs-r0015-b0003.bin
img-xcs-r0015-b0004.bin
img-xcs-r0015-b0005.bin
img-xcs-r0015-b0006.bin
img-xcs-r0015-b0007.bin
img-xcs-r0015-time.txt - list of time-record for all events in processed run.
img-xcs-r0015-time-ind.txt - list of time-record for all events in processed run with time index.
img-xcs-r0015-med.txt - file with metadata. In particular it has the original image size, number of image parts for splitting, number of images in run, etc.

Algorithms:

• The <int16_t> image data array is split for ordered number of equal parts and each part is saved in the output img-xcs-r0015-b000N.bin file sequentially for all selected events.
• The time record for each selected event is saved in the file img-xcs-r0015-time.txt.
• At the end of the splitting procedure:
  • the average time difference and its rms between sequential events is evaluated for all recorded time records.
  • The file img-xcs-r0015-time.txt is re-winded and for each record the time index is evaluated as unsigned value of

    <time-index> = (<event-time> + 0.5 <average-time-between-events>) /  <average-time-between-events> 
    

  • Event record with time index is saved in the file img-xcs-r0015-time.txt
• The file img-xcs-r0015-med.txt contains input parameters, image size, <average-time-between-events, maximal value of the time index etc.

Time correlation processing

ImgAlgos/test/corana application

Command:

<path>/corana -f <fname-data> [-t <fname-tau>] -h [-l <logfile>] [-b <basedir>]

For example the interactive and batch mode commands:

<path>/corana -f img-xcs-r0015-b0001.bin -t my-tau.txt 
bsub -q psnehq -o ~/<path-from-home-dir>/log.txt '<path>/corana -f img-xcs-r0015-b0000.bin'

Where ~/<path-from-home-dir>/ or <path> is arbitrary choice and is a matter of taste.

Produce files:

img-xcs-r0015-tau.txt          - string of tau values for which the auto-correlation function is evaluated
img-xcs-r0015-b0000-result.bin - auto-correlators for the part of the image for all tau values 
img-xcs-r0015-b0001-result.bin
img-xcs-r0015-b0002-result.bin
img-xcs-r0015-b0003-result.bin
img-xcs-r0015-b0004-result.bin
img-xcs-r0015-b0005-result.bin
img-xcs-r0015-b0006-result.bin
img-xcs-r0015-b0007-result.bin

Merging results

ImgAlgos/test/corana_merge application

Command:

<path>/corana_merge -f <fname-data> [-t <fname-tau>] -h [-l <logfile>] [-b <basedir>]

For example:

<path>/corana_merge -f img-xcs-r0015-b0001-result.bin -t my-tau.txt

Produce file:

img-xcs-r0015-image-result.bin

Wrapping script

./CorAnaSubmit.py

./CorAnaSubmit.py -c <config-file> -t <fname-tau> [-x] <xtc-file-list>

For example:

./CorAnaSubmit.py -c ana-misc-exp/psana-xcsi0112-r0015-img-auto-correlation.cfg -t my-tau.txt /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc

File formats

• img-xcs-r0015-b000N.bin:
  Currently it saves <uint16_t> amplitude for each pixel in binary format for:

  <data-for-img-partN-of-img1> <data-for-img-partN-of-img2> ... <data-for-img-partN-of-imgLast> 
  

• img-xcs-r0015-med.txt:

  IMAGE_ROWS      1300
  IMAGE_COLS      1340
  IMAGE_SIZE      1742000
  NUMBER_OF_FILES 8
  BLOCK_SIZE      217750
  REST_SIZE       0
  NUMBER_OF_IMGS  500
  FILE_TYPE       bin
  DATA_TYPE       uint16_t
  TIME_SEC_AVE    8.088413
  TIME_SEC_RMS    0.063639
  TIME_INDEX_MAX       499
  

• img-xcs-r0015-time.txt:

       1        0.000000  0.000000  20120616-080236.671607864    5366      0
       2        8.026429  8.026429  20120616-080244.698036743    8255      1
       3       16.144788  8.118359  20120616-080252.816395836   11177      2
       4       24.154835  8.010048  20120616-080300.826443448   14060      3
      ...
  

  where each record has:

  <image-in-file#> <t(sec)-from-the-1st-event> <dt(sec)> <time-stamp> <fiducials> <event#-since-configure>
  

• img-xcs-r0015-time.txt:

       1        0.000000  0.000000  20120616-080236.671607864    5366      0        0 
       2        8.026429  8.026429  20120616-080244.698036743    8255      1        1 
       3       16.144788  8.118359  20120616-080252.816395836   11177      2        2 
       4       24.154835  8.010048  20120616-080300.826443448   14060      3        3 
       5       32.281937  8.127102  20120616-080308.953545010   16985      4        4 
      ...
  

  where each record has:

  <image-in-file#>  <t(sec)-from-the-1st-event> <dt(sec)> <time-stamp> <fiducials> <event#-since-configure> <time-index-starting-from-0>
  

• img-xcs-r0015-b000N-result.bin:
  Currently it saves <float> correlator for each pixel in binary format for:

  <corr-for-img-partN-of-tau1> <corr-for-img-partN-of-tau2> ... <corr-for-img-partN-of-tauLast> 
  

• my-tau.txt:

   1 3 5 7 9 10 12 14 16 18 20 24 28 30 32 36 40 ... 160 180 200 240 280 300 320 360 400
  

  contains the tau values presented in terms of number of ordered images in the file.