This is a reference manual for algorithms implemented for the Integrated Data Processing Environment for Time Correlation Experiments. All work-flow can be split for main procedures:

  • Pre-processing,
  • Data processing, and
  • Presentation of results,

covered by this note with extensive references to the specific code.

Content

Pre-processing

Pre-processing algorithms consist of four procedures

  • Dark run scan,
  • Pedestals averaging,
  • Data scan, and
  • Data averaging,

described in this section.

Dark run scan

Defines for the dark run

  • the number of events
  • time records and average time interval

Submit command

psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-scan.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc

PSANA configuration script *peds-scan.cfg

[psana] 
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc 
#skip-events = IS_NOT_USED 
#events = FOR_ALL_EVENTS 
modules = ImgAlgos.ImgTimeStampList 

[ImgAlgos.ImgTimeStampList] 
print_bits = 13 
out_file = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-scan-tstamp-list.txt 
#======EOF======

PSANA module ImgAlgos.ImgTimeStampList

Pedestals averaging

Defines for dark run

  • pedestals
  • RMS
  • hot pixel mask

Submit command

psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc

PSANA configuration script *peds.cfg

[psana] 
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc 
skip-events = 0 
events = 75 
modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgAverage 

[ImgAlgos.Tahometer] 
print_bits = 7 

[ ImgAlgos.PrincetonImageProducer ] 
source = DetInfo(:Princeton) 
key_in = 
key_out = img 
print_bits = 1 

[ImgAlgos.ImgAverage] 
source = DetInfo(:Princeton) 
key = img 
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-ave.txt 
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-rms.txt 
hotpix_mask = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-hotpix-mask-thr-10.0ADU.txt 
hotpix_thr_adu = 10.0 
print_bits = 21

PSANA module ImgAlgos.PrincetonImageProducer

PSANA module ImgAlgos.ImgAverage

Data types:

  • Input image may be : double, float, int, uint16_t, uint8_t
  • Calculation: intensity in double, statistics in unsigned
  • Save arrays in txt for 
      save2DArrayInFile<double> ( m_aveFile, m_ave, m_rows, m_cols, m_print_bits & 16 );
  save2DArrayInFile<double> ( m_rmsFile, m_rms, m_rows, m_cols, m_print_bits & 16 );
  if (m_do_mask) save2DArrayInFile<int> ( m_hotFile, m_hot, m_rows, m_cols, m_print_bits & 16 );

PSANA module ImgAlgos.Tahometer

Data scan

Defines for data run

  • the number of events in the run
  • saves time records and average time interval
  • saves intensity monitor data

Submit command

psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc

PSANA configuration script *data-scan.cfg

[psana] 
#psana -m psana_examples.DumpPrinceton /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc 
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc 
modules = ImgAlgos.Tahometer ImgAlgos.ImgTimeStampList ImgAlgos.IntensityMonitorsData 
#skip-events = IS_NOT_USED 
#events = FOR_ALL_EVENTS 

[ImgAlgos.Tahometer] 
print_bits = 11 

[ImgAlgos.ImgTimeStampList] 
print_bits = 13 
out_file = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-tstamp-list.txt 

[ImgAlgos.IntensityMonitorsData] 
print_bits = 45 
#file_type = bin 
#file_data = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-data.txt 
file_data = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-data.txt 
file_header = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-comments.txt 
#======EOF======

PSANA module ImgAlgos.ImgTimeStampList

  • ImgAlgos.ImgTimeStampList.cpp
  • ImgAlgos.ImgTimeStampList.h
    Produces file with time records *data-scan-tstamp-list.txt : 
         0        0.000000  0.000000  20120616-080236.671607864    5366      0        0 
     1        8.026429  8.026429  20120616-080244.698036743    8255      1        1 
     2       16.144788  8.118359  20120616-080252.816395836   11177      2        2 
     3       24.154835  8.010048  20120616-080300.826443448   14060      3        3 
     4       32.281937  8.127102  20120616-080308.953545010   16985      4        4 
     5       40.400633  8.118696  20120616-080317.072241060   19907      5        5
...

PSANA module ImgAlgos.IntensityMonitorsData

IntensityMonitorsData produces files:

  • with headers for intensity monitor data data-scan-mons-comments.txt: 
    Heder for the data file: /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-data.txt
Number of sources: 5
Four values per source:
BldInfo(FEEGasDetEnergy) BldInfo(XCS-IPM-02) BldInfo(XCS-IPM-mono) DetInfo(XcsBeamline.1:Ipimb.4) DetInfo(XcsBeamline.1:Ipimb.5) 
Number of records in file: 500
  • with intensity monitor data data-scan-mons-data.txt: 
          0  2.19399 2.20273 0.05872 0.06004  0.62876 1.50444 0.60655 1.54488  0.00793 0.01758 0.00502 0.01968  0.00000 -0.00025 -0.00020 0.00000  0.00000 0.00000 0.00000 0.00000  
      1  2.10433 2.10410 0.06630 0.06703  0.66294 1.56792 0.62547 1.61347  0.00572 0.01147 0.00227 0.01366  0.00000 -0.00025 -0.00020 0.00000  0.00145 0.00610 0.00069 0.00153  
      2  2.40593 2.37730 0.06799 0.07501  0.69857 1.71524 0.70642 1.74614  0.00626 0.01590 0.00456 0.01946  0.00015 -0.00025 -0.00020 0.00000  0.00000 0.00000 0.00000 0.00000  
      3  2.44108 2.46661 0.07772 0.10092  0.69842 1.72280 0.71169 1.76384  0.00000 0.00064 -0.00040 0.00008  0.00000 -0.00025 -0.00020 0.00008  0.00000 0.00000 0.00000 0.00000  
      4  2.45779 2.45074 0.09505 0.08122  0.72374 1.71555 0.68254 1.74523  0.01320 0.03017 0.00952 0.03456  0.00000 -0.00025 -0.00020 0.00015  0.00000 0.00000 0.00000 0.00000  
      5  2.12671 2.13944 0.08422 0.07668  0.66477 1.61919 0.65263 1.63864  0.06844 0.15933 0.04858 0.17990  0.00000 -0.00025 -0.00020 0.00008  0.00000 0.00000 0.00114 0.00000  
...

Data averaging

Defines for data run

  • average image for selected range of events
  • evaluate saturated pixel mask

Submit command

psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-aver.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc

PSANA configuration script *data-aver.cfg

[psana] 
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc 
skip-events = 0 
events = 500 
modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgAverage ImgAlgos.ImgMaskEvaluation 

[ImgAlgos.Tahometer] 
print_bits = 7 

[ ImgAlgos.PrincetonImageProducer ] 
source = DetInfo(:Princeton) 
key_in = 
key_out = img 
print_bits = 1 

[ImgAlgos.ImgAverage] 
source = DetInfo(:Princeton) 
key = img 
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-ave.txt 
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-rms.txt 
print_bits = 25 
#evts_stage1 = 100 
#evts_stage2 = 100 
#gate_width1 = 200 
#gate_width2 = 50 

[ImgAlgos.ImgMaskEvaluation] 
source = DetInfo(:Princeton) 
key = img 
file_mask_satu = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-satpix-mask-level-65000ADU.txt 
file_mask_nois =  
file_mask_comb = 
file_frac_satu = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-satpix-frac-level-65000ADU.txt 
file_frac_nois =  
thre_satu = 65000 
frac_satu = 0 
dr_SoN_ave = 1 
thre_SoN = 5 
frac_nois = 0.05 
print_bits = 29 
#======EOF======

PSANA module ImgAlgos.ImgMaskEvaluation

The ImgAlgos.ImgMaskEvaluation produces the saturated pixel mask; the image size txt file *-satpix-mask-level-65000ADU.txt, consistiong 0(zero) or 1(one) for each bin.

Data processing

Data processing stage consists of three procedures:

  • Spliting,
  • Processing, and
  • Merging,
    described in this section.

Spliting

Submit command

psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-xcsi0112-r0015-split.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc

PSANA configuration script *split.cfg

[psana] 
# Command to run this script from release directory: 
# psana -c ImgAlgos/data/psana-split.cfg <path-to-xtc-file-name-pattern-for-one-run> 

#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc 
skip-events = 0 
events = 500 

modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgCalib ImgAlgos.ImgIntMonCorr ImgAlgos.ImgIntForBins ImgAlgos.ImgVsTimeSplitInFiles ImgAlgos.ImgAverage

[ImgAlgos.Tahometer] 
print_bits = 7 

[ ImgAlgos.PrincetonImageProducer ] 
source = DetInfo(:Princeton) 
key_in = 
key_out = img 
print_bits = 1 

[ImgAlgos.ImgCalib] 
source = DetInfo(:Princeton) 
key_in = img 
key_out = calibrated 
fname_peds = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-ave.txt 
fname_bkgd = 
fname_gain = 
fname_mask = 
fname_rms =  
threshold_nrms = 0 
do_threshold = true 
threshold = 20.0 
below_thre_value = 0 
print_bits = 5 

[ImgAlgos.ImgIntMonCorr] 
source = DetInfo(:Princeton) 
key_in = calibrated 
key_out = imon_corrected 
fname_imon_cfg = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-xcsi0112-r0015-imon-cfg.txt 
print_bits = 1 

[ImgAlgos.ImgIntForBins] 
source = DetInfo(:Princeton) 
key_in = imon_corrected 
fname_map_bins = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-map-static-q.txt 
fname_int_bins = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-int-static-q.txt 
number_of_bins = 1 
print_bits = 33 


[ImgAlgos.ImgVsTimeSplitInFiles] 
source = DetInfo(:Princeton) 
key = imon_corrected 
fname_prefix = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora 
file_type = bin 
add_tstamp = false 
ampl_thr = 20.0 
ampl_min = 1 
nfiles_out = 8 
print_bits = 29 


[ImgAlgos.ImgAverage] 
source = DetInfo(:Princeton) 
key = calibrated 
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-ave.txt 
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-rms.txt
#======EOF======

PSANA module ImgAlgos.ImgCalib

  • ImgAlgos.ImgCalib.cpp
  • ImgAlgos.ImgCalib.h
    This module is responsible for basic image intensity transformation. For correlation analysis it applys a couple of corrections:
  • subtract pedestals and
  • apply LLD threshold.

PSANA module ImgAlgos.ImgIntMonCorr

  • ImgAlgos.ImgIntMonCorr.cpp
  • ImgAlgos.ImgIntMonCorr.h
    Gets intensity monitor configuration file imon-cfg.txt produced in GUI: 
     BldInfo(FEEGasDetEnergy)         FEEGasDetEnergy  1 1 1 1   0 0     -1.0000   -1.0000    1.0000
 BldInfo(XCS-IPM-02)              XCS-IPM-02       1 1 1 1   0 0     -1.0000   -1.0000    1.0000
 BldInfo(XCS-IPM-mono)            XCS-IPM-mono     1 1 1 1   0 0     -1.0000   -1.0000   -1.0000
 DetInfo(XcsBeamline.1:Ipimb.4)   Ipimb.4          1 1 1 1   0 0     -1.0000   -1.0000    1.0000
 DetInfo(XcsBeamline.1:Ipimb.5)   Ipimb.5          1 1 1 1   0 0     -1.0000   -1.0000    1.0000

PSANA module ImgAlgos.ImgIntForBins

Gets the file *-map-static-q.txt with a map of static q bin numbers

Produces the file *-int-static-q.txt with intensities averaged over statis q bins:

       0  44.460 
       1  27.982 
       2  62.282 
       3  0.875 
       4  73.732 
       5  698.901 
       6  4.923 
       7  227.669 
       8  28.298 
...

PSANA module ImgAlgos.ImgVsTimeSplitInFiles

Gets/produces the file *-tau.txt with a list of tau intervals in terms of time(event) indexes, for example:

 1 2 3 4 5 6 7 8 9 10 12 14 16 20

Produces the file *-med.txt with metadata parameters from splitting algorithm:

IMAGE_ROWS      1300
IMAGE_COLS      1340
IMAGE_SIZE      1742000
NUMBER_OF_FILES 8
BLOCK_SIZE      217750
REST_SIZE       0
NUMBER_OF_IMGS  500
FILE_TYPE       bin
DATA_TYPE       f
TIME_SEC_AVE    8.088413
TIME_SEC_RMS    0.063639
TIME_INDEX_MAX       499

Processing

Submit command

corana -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000.bin -t ./tau-list.txt

Module ImgAlgos.corana

Module ImgAlgos.CorAnaData

Evaluation of correlators:

ImgAlgos/include/CorAna.h:  typedef float cor_t;

void CorAnaData::evaluateCorTau(unsigned tau) // tau in number of frames between images
{
  m_log << "\nCorAnaData::evaluateCorTau(tau): tau=" << tau;
  std::fill_n(m_sum_gi, m_blk_size, double(0));
  std::fill_n(m_sum_gf, m_blk_size, double(0));
  std::fill_n(m_sum_g2, m_blk_size, double(0));
  std::fill_n(m_sum_st, m_blk_size, unsigned(0));
  std::fill_n(m_cor_gi, m_blk_size, cor_t(0));
  std::fill_n(m_cor_gf, m_blk_size, cor_t(0));
  std::fill_n(m_cor_g2, m_blk_size, cor_t(0));

  for (unsigned ti=0; ti<m_tind_size-tau; ti++) {
       unsigned tf=ti+tau;

       if ( ! (tf<m_tind_size) ) break;
       
       // get the event index in array for time index
       int evi = m_tind_to_evind[ti];
       int evf = m_tind_to_evind[tf];

       // If the event does not exist for specified time index -> skip it in sum
       if(evi<0) continue;
       if(evf<0) continue;

       sumCorTau((unsigned)evi,(unsigned)evf);
  }
}

//----------------
void CorAnaData::sumCorTau(unsigned i, unsigned f) // i and f are the event indexes
{
   data_t* p_i = &m_data[i*m_blk_size];
   data_t* p_f = &m_data[f*m_blk_size];
   double Ii, If;

   for(unsigned pix=0; pix<m_blk_size; pix++) {
     Ii = p_i[pix]; 
     If = p_f[pix]; 
     m_sum_gi[pix] += Ii; 
     m_sum_gf[pix] += If; 
     m_sum_g2[pix] += Ii*If; 
     m_sum_st[pix] ++; 
   }
}

Averaging:

   for(unsigned pix=0; pix<m_blk_size; pix++) {
       if(m_sum_st[pix]<1) continue;
       m_cor_gi[pix] = cor_t( m_sum_gi[pix] / m_sum_st[pix] ); 
       m_cor_gf[pix] = cor_t( m_sum_gf[pix] / m_sum_st[pix] ); 
       m_cor_g2[pix] = cor_t( m_sum_g2[pix] / m_sum_st[pix] ); 
   }

Module ImgAlgos.CorAna

Module ImgAlgos.CorAnaInputParameters

Merging

Submit command

corana_merge -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000-result.bin -t /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-tau.txt

Module ImgAlgos.corana_merge

Module ImgAlgos.CorAnaMergeFiles

Presentation of results

Presentation of results is implemented in the GUIViewResults.py, which basically is a panel of buttons for different plots.

Summary

In this note we presented a functional description of implemented procedures for time correlation analysis. All modules have convenient reference to the latest version of the code.

References

  • No labels