Page History
Content
...
This is a reference manual for algorithms implemented for the Integrated Data Processing Environment for Time Correlation Experiments. All work-flow can be split for main procedures:
- Pre-processing
...
Dark run scan scan
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc
|
PSANA configuration script *peds.cfg
- ,
- Data processing, and
- Presentation of results,
covered by this note with extensive references to the specific code.
Content
| Table of Contents |
|---|
Pre-processing
Pre-processing algorithms consist of four procedures
- Dark run scan,
- Pedestals averaging,
- Data scan, and
- Data averaging,
described in this section.
Dark run scan
Defines for the dark run
- the number of events
- time records and average time interval
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-scan.cfg /reg/d/ana12/xcs/xcsi0112/ |
| Code Block |
[psana] #files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc skip-events = 0 events = 75 modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgAverage |
PSANA configuration script *peds-scan.cfg
| Code Block |
|---|
[psana] #files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0020-*.xtc #skip-events = IS_NOT_USED #events = FOR_ALL_EVENTS modules = ImgAlgos.ImgTimeStampList [ImgAlgos.TahometerImgTimeStampList] print_bits = 713 [ ImgAlgos.PrincetonImageProducer ] source = DetInfo(:Princeton) key_in = key_out = img print_bits = 1 [ImgAlgos.ImgAverage] source =out_file = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-scan-tstamp-list.txt #======EOF====== |
PSANA module ImgAlgos.ImgTimeStampList
Pedestals averaging
Defines for dark run
- pedestals
- RMS
- hot pixel mask
Submit command
| Code Block |
|---|
psana -c /reg/neh/ DetInfo(:Princeton) key = img avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-ave.txt rmsfile =.cfg /reg/nehd/home1ana12/dubrovinxcs/LCLSxcsi0112/PSANA-V01/work/t1-xcsi0112xtc/e167-r0020-*.xtc |
PSANA configuration script *peds.cfg
| Code Block |
|---|
[psana] #filespeds-rms.txt hotpix_mask = /reg/nehd/home1ana12/dubrovinxcs/LCLSxcsi0112/PSANA-V01/work/t1-xcsi0112xtc/e167-r0020-hotpix-mask-thr-10.0ADU.txt hotpix_thr_adu*.xtc skip-events = 10.0 print_bitsevents = 75 modules 21= |
...
ImgAlgos. |
...
Tahometer ImgAlgos.PrincetonImageProducer |
...
PSANA module ImgAlgos.ImgAverage
Data types:
- Input image may be : double, float, int, uint16_t, uint8_t
- Calculation: intensity in double, statistics in unsigned
- Save arrays in txt for
Code Block save2DArrayInFile<double> ( m_aveFile, m_ave, m_rows, m_cols, m_print_bits & 16 ); save2DArrayInFile<double> ( m_rmsFile, m_rms, m_rows, m_cols, m_print_bits & 16 ); if (m_do_mask) save2DArrayInFile<int> ( m_hotFile, m_hot, m_rows, m_cols, m_print_bits & 16 );
Averaging in ImgAlgos::ImgAverage::
| Code Block |
|---|
template <typename T>
void accumulateCorrelators(const T* data)
{
double amp(0);
for (unsigned i=0; i<m_size; ++i) {
amp = (double)data[i];
if ( m_gate_width > 0 && std::abs(amp-m_ave[i]) > m_gate_width ) continue;
m_stat[i] ++;
m_sum [i] += amp;
m_sum2[i] += amp*amp;
}
}
|
Averaging in ImgAlgos::ImgAverage::procStatArrays()
| Code Block |
|---|
for (unsigned i=0; i!=m_size; ++i) {
double stat = m_stat[i];
if (stat > 0) {
double ave = m_sum[i] / stat;
m_ave[i] = ave;
m_rms[i] = std::sqrt(m_sum2[i] / stat - ave*ave);
}
else
{
m_ave[i] = 0;
m_rms[i] = 0;
}
}
|
Hot pixel mask in ImgAlgos::ImgAverage::procStatArrays()
| Code Block |
|---|
if (m_do_mask) {
for (unsigned i=0; i!=m_size; ++i)
m_hot[i] = (m_rms[i] > m_hot_thr) ? 0 : 1;
}
|
Data scan
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-aver.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
|
Saturated pixel mask
...
Time records
...
Intensity monitor records
...
Data processing
Split
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-xcsi0112-r0015-split.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
|
PSANA configuration script *split.cfg
...
ImgAlgos.ImgAverage
[ImgAlgos.Tahometer]
print_bits = 7
[ ImgAlgos.PrincetonImageProducer ]
source = DetInfo(:Princeton)
key_in =
key_out = img
print_bits = 1
[ImgAlgos.ImgAverage]
source = DetInfo(:Princeton)
key = img
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-ave.txt
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-rms.txt
hotpix_mask = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-hotpix-mask-thr-10.0ADU.txt
hotpix_thr_adu = 10.0
print_bits = 21 |
PSANA module ImgAlgos.PrincetonImageProducer
PSANA module ImgAlgos.ImgAverage
Data types:
- Input image may be : double, float, int, uint16_t, uint8_t
- Calculation: intensity in double, statistics in unsigned
- Save arrays in txt for
Code Block save2DArrayInFile<double> ( m_aveFile, m_ave, m_rows, m_cols, m_print_bits & 16 ); save2DArrayInFile<double> ( m_rmsFile, m_rms, m_rows, m_cols, m_print_bits & 16 ); if (m_do_mask) save2DArrayInFile<int> ( m_hotFile, m_hot, m_rows, m_cols, m_print_bits & 16 );
PSANA module ImgAlgos.Tahometer
Data scan
Defines for data run
- the number of events in the run
- saves time records and average time interval
- saves intensity monitor data
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
|
PSANA configuration script *data-scan.cfg
| Code Block |
|---|
[psana]
#psana -m psana_examples.DumpPrinceton /reg/d/psdm/XCS/xcsi0112/xtc/e167-r0015-s00-c00.xtc
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
modules = ImgAlgos.Tahometer ImgAlgos.ImgTimeStampList ImgAlgos.IntensityMonitorsData
#skip-events = IS_NOT_USED
#events = FOR_ALL_EVENTS
[ImgAlgos.Tahometer]
print_bits = 11
[ImgAlgos.ImgTimeStampList]
print_bits = 13
out_file = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-tstamp-list.txt
[ImgAlgos.IntensityMonitorsData]
print_bits = 45
#file_type = bin
#file_data = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-data.txt
file_data = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-data.txt
file_header = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-scan-mons-comments.txt
#======EOF======
|
PSANA module ImgAlgos.ImgTimeStampList
- ImgAlgos.ImgTimeStampList.cpp
- ImgAlgos.ImgTimeStampList.h
Produces file with time records*data-scan-tstamp-list.txt:Code Block 0 0.000000 0.000000 20120616-080236.671607864 5366 0 0 1 8.026429 8.026429 20120616-080244.698036743 8255 1 1 2 16.144788 8.118359 20120616-080252.816395836 11177 2 2 3 24.154835 8.010048 20120616-080300.826443448 14060 3 3 4 32.281937 8.127102 20120616-080308.953545010 16985 4 4 5 40.400633 8.118696 20120616-080317.072241060 19907 5 5 ...
PSANA module ImgAlgos.IntensityMonitorsData
IntensityMonitorsData produces files:
- with headers for intensity monitor data
data-scan-mons-comments.txt:Code Block Heder for the data file: /reg/neh/home1/
...
dubrovin/LCLS/PSANA-V01/work/t1
...
-xcsi0112-r0015-
...
data-scan-mons-data.txt
...
...
Number
...
of
...
sources:
...
5 Four values per source: BldInfo(FEEGasDetEnergy) BldInfo(XCS-IPM-02) BldInfo(XCS-IPM-mono) DetInfo(XcsBeamline.1:Ipimb.4) DetInfo(XcsBeamline.1:Ipimb.5) Number of records in file: 500- with intensity monitor data
data-scan-mons-data.txt:Code Block 0 2.19399 2.20273 0.05872 0.06004 0.62876 1.50444 0.60655 1.54488 0.00793 0.01758 0.00502 0.01968 0.00000 -0.00025 -0.00020 0.00000 0.00000 0.00000 0.00000 0.00000 1 2.10433 2.10410 0.06630 0.06703 0.66294 1.56792 0.62547 1.61347 0.00572 0.01147 0.00227 0.01366 0.00000 -0.00025 -0.00020 0.00000 0.00145 0.00610 0.00069 0.00153 2 2.40593 2.37730 0.06799 0.07501 0.69857 1.71524 0.70642 1.74614 0.00626 0.01590 0.00456 0.01946 0.00015 -0.00025 -0.00020 0.00000 0.00000 0.00000 0.00000 0.00000 3 2.44108 2.46661 0.07772 0.10092 0.69842 1.72280 0.71169 1.76384 0.00000 0.00064 -0.00040 0.00008 0.00000 -0.00025 -0.00020 0.00008 0.00000 0.00000 0.00000 0.00000 4 2.45779 2.45074 0.09505 0.08122 0.72374 1.71555 0.68254 1.74523 0.01320 0.03017 0.00952 0.03456 0.00000 -0.00025 -0.00020 0.00015 0.00000 0.00000 0.00000 0.00000 5 2.12671 2.13944 0.08422 0.07668 0.66477 1.61919 0.65263 1.63864 0.06844 0.15933 0.04858 0.17990 0.00000 -0.00025 -0.00020 0.00008 0.00000 0.00000 0.00114 0.00000 ...
Data averaging
Defines for data run
- average image for selected range of events
- evaluate saturated pixel mask
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-aver.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
|
PSANA configuration script *data-aver.cfg
| Code Block |
|---|
[psana]
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
skip-events = 0
events = 500
modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgAverage ImgAlgos.ImgMaskEvaluation
[ImgAlgos.Tahometer]
print_bits = 7
[ ImgAlgos.PrincetonImageProducer ]
source = DetInfo(:Princeton)
key_in =
key_out = img
print_bits = 1
[ImgAlgos.ImgAverage]
source = DetInfo(:Princeton)
key = img
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-ave.txt
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-rms.txt
print_bits = 25
#evts_stage1 = 100
#evts_stage2 = 100
#gate_width1 = 200
#gate_width2 = 50
[ImgAlgos.ImgMaskEvaluation]
source = DetInfo(:Princeton)
key = img
file_mask_satu = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-satpix-mask-level-65000ADU.txt
file_mask_nois =
file_mask_comb =
file_frac_satu = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-satpix-frac-level-65000ADU.txt
file_frac_nois =
thre_satu = 65000
frac_satu = 0
dr_SoN_ave = 1
thre_SoN = 5
frac_nois = 0.05
print_bits = 29
#======EOF======
|
PSANA module ImgAlgos.ImgMaskEvaluation
The ImgAlgos.ImgMaskEvaluation produces the saturated pixel mask; the image size txt file *-satpix-mask-level-65000ADU.txt, consistiong 0(zero) or 1(one) for each bin.
Data processing
Data processing stage consists of three procedures:
- Spliting,
- Processing, and
- Merging,
described in this section.
Spliting
Submit command
| Code Block |
|---|
psana -c /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-xcsi0112-r0015-split.cfg /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
|
PSANA configuration script *split.cfg
| Code Block |
|---|
[psana]
# Command to run this script from release directory:
# psana -c ImgAlgos/data/psana-split.cfg <path-to-xtc-file-name-pattern-for-one-run>
#files = /reg/d/ana12/xcs/xcsi0112/xtc/e167-r0015-*.xtc
skip-events = 0
events = 500
modules = ImgAlgos.Tahometer ImgAlgos.PrincetonImageProducer ImgAlgos.ImgCalib ImgAlgos.ImgIntMonCorr ImgAlgos.ImgIntForBins ImgAlgos.ImgVsTimeSplitInFiles ImgAlgos.ImgAverage
[ImgAlgos.Tahometer]
print_bits = 7
[ ImgAlgos.PrincetonImageProducer ]
source = DetInfo(:Princeton)
key_in =
key_out = img
print_bits = 1
[ImgAlgos.ImgCalib]
source = DetInfo(:Princeton)
key_in = img
key_out = calibrated
fname_peds = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0020-peds-ave.txt
fname_bkgd =
fname_gain =
fname_mask =
fname_rms =
threshold_nrms = 0
do_threshold = true
threshold = 20.0
below_thre_value = 0
print_bits = 5
[ImgAlgos.ImgIntMonCorr]
source = DetInfo(:Princeton)
key_in = calibrated
key_out = imon_corrected
fname_imon_cfg = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-xcsi0112-r0015-imon-cfg.txt
print_bits = 1
[ImgAlgos.ImgIntForBins]
source = DetInfo(:Princeton)
key_in = imon_corrected
fname_map_bins = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-map-static-q.txt
fname_int_bins = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-int-static-q.txt
number_of_bins = 1
print_bits = 33
[ImgAlgos.ImgVsTimeSplitInFiles]
source = DetInfo(:Princeton)
key = imon_corrected
fname_prefix = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora
file_type = bin
add_tstamp = false
ampl_thr = 20.0
ampl_min = 1
nfiles_out = 8
print_bits = 29
[ImgAlgos.ImgAverage]
source = DetInfo(:Princeton)
key = calibrated
avefile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-ave.txt
rmsfile = /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-xcsi0112-r0015-data-rms.txt
#======EOF======
|
PSANA module ImgAlgos.ImgCalib
- ImgAlgos.ImgCalib.cpp
- ImgAlgos.ImgCalib.h
This module is responsible for basic image intensity transformation. For correlation analysis it applys a couple of corrections: - subtract pedestals and
- apply LLD threshold.
PSANA module ImgAlgos.ImgIntMonCorr
- ImgAlgos.ImgIntMonCorr.cpp
- ImgAlgos.ImgIntMonCorr.h
Gets intensity monitor configuration fileimon-cfg.txtproduced in GUI:Code Block BldInfo(FEEGasDetEnergy) FEEGasDetEnergy 1 1 1 1 0 0 -1.0000 -1.0000 1.0000 BldInfo(XCS-IPM-02) XCS-IPM-02 1 1 1 1 0 0 -1.0000 -1.0000 1.0000 BldInfo(XCS-IPM-mono) XCS-IPM-mono 1 1 1 1 0 0 -1.0000 -1.0000 -1.0000 DetInfo(XcsBeamline.1:Ipimb.4) Ipimb.4 1 1 1 1 0 0 -1.0000 -1.0000 1.0000 DetInfo(XcsBeamline.1:Ipimb.5) Ipimb.5 1 1 1 1 0 0 -1.0000 -1.0000 1.0000
PSANA module ImgAlgos.ImgIntForBins
Gets the file *-map-static-q.txt with a map of static q bin numbers
Produces the file *-int-static-q.txt with intensities averaged over statis q bins:
| Code Block |
|---|
0 44.460
1 27.982
2 62.282
3 0.875
4 73.732
5 698.901
6 4.923
7 227.669
8 28.298
...
|
PSANA module ImgAlgos.ImgVsTimeSplitInFiles
Gets/produces the file *-tau.txt with a list of tau intervals in terms of time(event) indexes, for example:
| Code Block |
|---|
1 2 3 4 5 6 7 8 9 10 12 14 16 20
|
Produces the file *-med.txt with metadata parameters from splitting algorithm:
| Code Block |
|---|
IMAGE_ROWS 1300
IMAGE_COLS 1340
IMAGE_SIZE 1742000
NUMBER_OF_FILES 8
BLOCK_SIZE 217750
REST_SIZE 0
NUMBER_OF_IMGS 500
FILE_TYPE bin
DATA_TYPE f
TIME_SEC_AVE 8.088413
TIME_SEC_RMS 0.063639
TIME_INDEX_MAX 499
|
Processing
Submit command
| Code Block |
|---|
corana -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000.bin -t ./tau-list.txt
|
Module ImgAlgos.corana
Module ImgAlgos.CorAnaData
Evaluation of correlators:
| Code Block |
|---|
ImgAlgos/include/CorAna.h: typedef float cor_t;
void CorAnaData::evaluateCorTau(unsigned tau) // tau in number of frames between images
{
m_log << "\nCorAnaData::evaluateCorTau(tau): tau=" << tau;
std::fill_n(m_sum_gi, m_blk_size, double(0));
std::fill_n(m_sum_gf, m_blk_size, double(0));
std::fill_n(m_sum_g2, m_blk_size, double(0));
std::fill_n(m_sum_st, m_blk_size, unsigned(0));
std::fill_n(m_cor_gi, m_blk_size, cor_t(0));
std::fill_n(m_cor_gf, m_blk_size, cor_t(0));
std::fill_n(m_cor_g2, m_blk_size, cor_t(0));
for (unsigned ti=0; ti<m_tind_size-tau; ti++) {
unsigned tf=ti+tau;
if ( ! (tf<m_tind_size) ) break;
// get the event index in array for time index
int evi = m_tind_to_evind[ti];
int evf = m_tind_to_evind[tf];
// If the event does not exist for specified time index -> skip it in sum
if(evi<0) continue;
if(evf<0) continue;
sumCorTau((unsigned)evi,(unsigned)evf);
}
}
//----------------
void CorAnaData::sumCorTau(unsigned i, unsigned f) // i and f are the event indexes
{
data_t* p_i = &m_data[i*m_blk_size];
data_t* p_f = &m_data[f*m_blk_size];
double Ii, If;
for(unsigned pix=0; pix<m_blk_size; pix++) {
Ii = p_i[pix];
If = p_f[pix];
m_sum_gi[pix] += Ii;
m_sum_gf[pix] += If;
m_sum_g2[pix] += Ii*If;
m_sum_st[pix] ++;
}
}
|
Averaging:
| Code Block |
|---|
for(unsigned pix=0; pix<m_blk_size; pix++) {
if(m_sum_st[pix]<1) continue;
m_cor_gi[pix] = cor_t( m_sum_gi[pix] / m_sum_st[pix] );
m_cor_gf[pix] = cor_t( m_sum_gf[pix] / m_sum_st[pix] );
m_cor_g2[pix] = cor_t( m_sum_g2[pix] / m_sum_st[pix] );
}
|
Module ImgAlgos.CorAna
Module ImgAlgos.CorAnaInputParameters
Merging
Submit command
| Code Block |
|---|
corana_merge -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000-result.bin -t /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-tau.txt
|
Module ImgAlgos.corana_merge
Module ImgAlgos.CorAnaMergeFiles
Presentation of results
Presentation of results is implemented in the GUIViewResults.py, which basically is a panel of buttons for different plots.
Summary
In this note we presented a functional description of implemented procedures for time correlation analysis. All modules have convenient reference to the latest version of the code.
References
- Module
ImgAlgos.GlobalMethods - ImgAlgos.GlobalMethods.cpp
- ImgAlgos.GlobalMethods.h
- Modules of
ImgAlgos - ImgAlgos.*.cpp
- ImgAlgos.*.h
PSANA module ImgAlgos.ImgCalib
PSANA module ImgAlgos.ImgIntMonCorr
PSANA module ImgAlgos.ImgIntForBins
PSANA module ImgAlgos.ImgVsTimeSplitInFiles
Processing
Submit command
| Code Block |
|---|
corana -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000.bin -t ./tau-list.txt
|
Module ImgAlgos.corana
Module ImgAlgos.CorAnaData.cpp
Module ImgAlgos.CorAna.cpp
Module ImgAlgos.CorAnaInputParameters
Merging
Submit command
| Code Block |
|---|
corana_merge -f /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-b0000-result.bin -t /reg/neh/home1/dubrovin/LCLS/PSANA-V01/work/t1-cora-r0015-tau.txt
|
Module ImgAlgos.corana_merge
PSANA module ImgAlgos.CorAnaMergeFiles
Presentation of results
<!--
Module ImgAlgos.
...
Overview
Content Tools