...

Test of the NDArrCalib module for pnCCD.

Module ImgAlgos::

...

ImgAverage

This module averages over events the per-pixel data of the image array (ndarray<const double,2>) and saves files for averaged, rms values, and, if requested, the hot pixel mask. Input data can be specified by the source and key parameters. Averaging may have up to three stages, depending on configuration parameters:

• 0-stage: the pixel amplitudes are averaged without any constrains for events from 0 to evts_stage1, the preliminary averaged and rms values are defined for each pixel at the end of this stage.
• 1-stage: starting from event evts_stage1 the pixel data are collected only for abs(amplitude-average0) < gate_width1. At the end of this stage the preliminary averaged and rms values are defined for each pixel.
• 2-stage: starting from the event evts_stage1 + evts_stage2 the pixel data are collected only for abs(amplitude-average1) < gate_width2. At the end of this stage the preliminary averaged and rms values are defined for each pixel and saved in the files specified by the avefile and rmsfile parameters, respectively.
  This 3-stage averaging algorithm eliminates large statistical fluctuations in the pixel amplitude spectrum.

If all file names are empty (by default), the files with pre-defined names "img-ave-r####.dat" and "img-rms-r####.dat" (where #### stands for run number) will be saved for averaged and rms images, respectively. Otherwise, the files with specified names will be saved.
Default parameters are set for regular single-stage averaging without any constrains.
See also Example for Module ImgAlgos::ImgAverage.

Module ImgAlgos::ImgMaskEvaluation

This module gets the image data array (ndarray<const T,2>), where T stands for double, float, int, uint8_t, or uint16_t, and evaluates two masks:

1. saturated mask for pixels, which had an intensity above the saturation-threshold with frequency grater than specified fraction of events.
2. noise mask for pixels, which estimated S/N ratio exceeds the S/N-threshold with frequency grater than specified fraction of events.

In the endJob this module saves files (if their names are provided) of image size for:

1. saturated mask
2. noise mask,
3. combined mask,
4. fraction of noisy events,
5. fraction of saturated events.

The S/N is estimated by averaging over neigbouring pixels.

See also Example for Module ImgAlgos::ImgMaskEvaluation.

Module ImgAlgos::ImgCalib

ImgCalib gets the raw image from data and process it as follows:

1. subtracts pedestals,
2. subtracts normalized background,
3. apply gain factors,
4. apply mask, and
5. apply threshold as a constant low level,
6. apply threshold as N*RMS,
   and saves the corrected image in the event.

Functionality:

• ImgCalib uses the source and key_in parameters to get the input raw image (as ndarray<const T,2> object), where T stands for int16_t(from V00-03-46), uint16_t, int, float, uint8_t, or double,
• gets the calibration parameters from files fname_peds, fname_bkgd, fname_gain, fname_mask, and fname_rms, if their names are specified,

• the specified by the file name corrections are applied per-pixel to raw data image as follows:

  Code Block
  bgColor #F7F7ED
  A_cor = A_raw (1) - pedestal | if the file name is specified in the parameter "fname_peds" (2) - N*background | if the file name is specified in the parameter "fname_bkgd" (3) * gain | if the file name is specified in the parameter "fname_gain" (4) apply mask | if the file name is specified in the parameter "fname_mask" (5) apply N*RMS threshold | if the file name is specified in the parameter "fname_nrms" (6) apply threshold | if the "do_threshold" = true

• corrected image is saved in the event with key key_out as double type.

Details:

• All files with input calibration parameters should have the same shape as image and formatted as regular text file containing 2d matrix (table) of float (or integer) values, with columns separated by space(s), ' ', and rows(lines) terminated by the '\n'.
• Background normalization is performed in window defined by the parameters bkgd_row_min, bkgd_row_max, bkgd_col_min, and bkgd_col_max. Normalization factor N is evaluated for pixel amplitudes in this window as:
  N = sum(A_raw - pedestal) / sum(A_bkgd).
• Masking algorithm assumes that good pixels in the fname_mask file should be marked by '1' (ones) and bad pixels – by '0' (zeros). Mask is applied as a last correction. The bad masked pixel amplitudes are substituted by the masked_value.

The pedestal, background, gain, mask, and N*RMS corrections are applied if associated file name is specified.
The constant low level threshold is applied if do_threshold is true.
Corrections are not applied by default or with empty file names.

See also Example for Module ImgAlgos::ImgCalib.

Module ImgAlgos::ImgIntForBins

Functionality:

• ImgIntForBins uses the source and key_in parameters to get the input image (as ndarray<const T,2> object), where T stands for uint16_t, int, float, uint8_t, or double,
• gets the pixel-bin indexes from files fname_map_bins, which has a size of image,
• calculates average per pixel intensity for each of number_of_bins bin (from 0 to number_of_bins-1),
• saves the 2-D array of <I>(event, bin) in file fname_int_bins.

Module ImgAlgos::ImgRadialCorrection

This module averages the image pixel amplitude in r-phi bins, normalizes it per single pixel and subtracts the average amplitude from each pixel. Image is obtained from event by its source and inkey values. The output corrected image is saved in the event with outkey keyword. Carthesian to polar coordinate transformation is done with respect to center coordinates xcenter, ycenter. The central region (r<rmin) and edges of the image (r>rmax) pixels can be removed from further consideration by setting rmin and rmax. The number of radial bins is defined as an int(rmax-rmin). The number of angular bins is set by n_phi_bins.

See also Example for Module ImgAlgos::ImgRadialCorrection.

Module ImgAlgos::ImgPixAmpFilter

The ImgAlgos::ImgPixAmpFilter is a filter for event selection.
This filter counts the number of image pixels in the specified window with amplitude exceeding the threshold. If the number of high-amplitude pixels exceed the numPixMin, the event is passed for further analysis.

The algorithm performance was tested for CSPad images. This algorithm consumes up to 15 ms/event on psana0205 for full CSPad (1650x1650) window size. For smaller window consumed time is negligible, comparing to the image reconstruction time, which is ~70 ms/event (for cspad_mod.CsPadCalib and CSPadPixCoords::CSPadImageProducer) on psana0205.

Remarks:

• The default key ("Image2D") stands for the CSPadPixCoords::Image2D<double> image object. Other key names work for the ndarray<const double,2> image object.
• The xmin, xmax, ymin, ymax (in pixels) defines the window in the image for pixel counting. Default values means the entire image range.
• The threshold, numPixMin, and the window extents have to be adjusted for particular experiment.

Module ImgAlgos::ImgPeakFinder

This algorithm was motivated by users of amo42112:
1. Select the pixels in the window xmin, xmax, ymin, ymax
with amplitudes above the threshold_low.
2. Optionally smears image for selected pixels, using 2-d matrix of weights over pixels from -smear_radius to +smear_radius around each smeared pixel amplitude. The matrix of weights is defined by the 2-d Gaussian function of width sigma. If sigma=0 smearing is not applied.
3. Find peaks as pixels with absolute-maximal amplitude above the threshold_high in the center of the matrix -peak_radius to +peak_radius.
4. Put the vector or ndarray of found peaks in the event with key peaksKey or peaks_nda respectively. Each entry of this vector has an object of the struct Peak, containing x, y positions, peak pixel amplitude, the total amplitude in the matrix, defined by the peak_radius, and the number of pixels in the matrix above threshold_low:

struct Peak{
   double x;
   double y;
   double ampmax;  // amplitude in the peak maximum
   double amptot;  // total amplitude in the range of {{peak_radius}}
   unsigned npix;  // number of pixels in the range of {{peak_radius}}
} ;

Remarks:

• This algorithm consumes ~15 ms/event on psana0101 for full Opal1000 (1024x1024) camera image.
• Smearing algorithm use a "safety margin" which is currently set to 10 pixels (offset from each boarder of the full image size).
• Since V00-03-58 saves table of peaks as  ndarray<float,2> with shape=[npeaks,5] if key peaks_nda is non-empty.

See also Example for Module ImgAlgos::ImgPeakFinder.

Module ImgAlgos::ImgPeakFilter

This module use results and should work after the ImgAlgos::ImgPeakFinder. It gets the vector of peaks for the source and key, loops over all founded peaks and counts the number of peaks above the thresholds threshold_peak and threshold_total. If the selection_mode is "SELECTION_ON" and the number of found peaks exceeds the n_peaks_min the event is passed for further analysis/processing,
the table of found peaks may be saved in file with prefix defined by the fname parameter.

See also Example for Module ImgAlgos::ImgPeakFilter.

Module ImgAlgos::ImgPeakFinderAB

This module finds peaks in the ndarray<const double,2> image object. Algorithm and the list of parameters are the same as described is section Module ImgAlgos::CSPadArrPeakFinder. The only difference is that the image size is defined by the ndarray<const double,2> object.

See also Example for Module ImgAlgos::ImgPeakFinderAB.

Module ImgAlgos::ImgHitFinder

ImgHitFinder is created by request for amo74213. It works pretty similar to ImgCalib, but the threshold algorithms are extended and background subtraction is removed.
It gets the raw image from data and process it as follows:

1. subtracts pedestals,
2. apply gain factors,
3. apply mask, and
4. apply one of the threshold algorithms
   and saves the corrected image or hit-pixel map in the event.

Functionality:

• ImgHitFinder uses the source and key_in parameters to get the input raw image (as ndarray<const T,2> object), where T stands for uint16_t, int, float, uint8_t, or double,
• gets the calibration parameters from files fname_peds, fname_gain, fname_mask, and fname_thre, if their names are specified,

• the specified by the file name corrections are applied per-pixel to raw data image as follows:

  Code Block
  bgColor #F7F7ED
  A_cor = A_raw (1) - pedestal | if the file name is specified in the parameter "fname_peds" (2) * gain | if the file name is specified in the parameter "fname_gain" (3) apply mask | if the file name is specified in the parameter "fname_mask" (4) apply threshold | if the file name is specified in the parameter "fname_thre"

• corrected image is saved in the event with key key_out as double (or unsigned for pixel map) type.

Details:

• All files with input calibration parameters should have the same shape as image and formatted as regular text file containing 2d matrix (table) of float (or integer) values, with columns separated by space(s), ' ', and rows(lines) terminated by the '\n'.
• Masking algorithm assumes that good pixels in the fname_mask file should be marked by '1' (ones) and bad pixels – by '0' (zeros). The bad masked pixel amplitudes are substituted by the masked_value.

Corrections are not applied by default or with empty file names.

See also Example for Module ImgAlgos::ImgHitFinder.

Module ImgAlgos::ImgSpectra

This module is motivated by the discussion with Josef Frisch, Ryan Coffee, Nick Hartmann. In xppi0412 etc. experiments they need to extract two spectra from Opal1000 camera image for signal and reference, evaluate their relative difference, and find peak position in the differential spectrum.
Module mgAlgos::ImgSpectra works as follows:

• gets the ndarray<const double,2> image object from event,
• selects two spectral band regions and integrates amplitudes for each column (it is assumed that both spectra are oriented along the rows),
• saves two spectral arrays for signal and reference bands and their relative difference as another ndarray<const double,2> object with shape (3,<number-of-columns>) in the event.
• Further analytical work is assumed to be done in the next module ImgAlgos::ImgSpectraProc.

See also Example for Module ImgAlgos::ImgSpectra.

Module ImgAlgos::ImgSpectraProc

Works after module ImgAlgos::ImgSpectra.
This module is designed as an example, in order to show how to get access to spectral array.
In particular, method ImgSpectraProc::getSpectra(...) shows how to get pointer to data, and method ImgSpectraProc::printSpectra(...) iterates over array and selectively prints its elements.

See also Example for Module ImgAlgos::ImgSpectraProc.

Module ImgAlgos::ImgSaveInFile

Module ImgSaveInFile receives from the event the image object using source and key parameters and saves it in the ftype format with prefix file name fname for event(s) specified by the parameters eventSave or saveAll. Currently implemented file formats: txt, bin, tiff, and png.

ImgSaveInFile works after the CSPadImageProducer, CSPadInterpolImageProducer, CameraImageProducer, PnccdImageProducer, etc., which produce image object in formats CSPadPixCoords::Image2D<T> or ndarray<const T,2>, where the T stands for one of the data types, double, float, int, uint8_t, or uint16_t.

See also Example for Module ImgAlgos::ImgSaveInFile

Module ImgAlgos::ImgVsTimeSplitInFiles

This module is a part of complex algorithm, described in Command Line Interface For Time Correlation Analysis.

This module is designed for parallel image processing for correlation analysis.
Functionality:

• get image for each event as an ndarray<const T,2> object,
• splits it for nfiles_out equal parts,
• saves each part of the image for all events in the job in a separate file with name <fname_prefix>-<fname-common>-b<block-number>.<file_type>,
• saves metadata in the text file with name: <fname_prefix>-<fname-common>-med.txt,
• saves counter number and the time records in file: <fname_prefix>-<fname-common>-time.txt.

See also Example for Module ImgAlgos::ImgVsTimeSplitInFiles.

Further processing of the files

• <fname_prefix>-<fname-common>-b<block-number>.<file_type>
• <fname_prefix>-<fname-common>-med.txt

• <fname_prefix>-<fname-common>-time.txt
  is implemented in stand-alone c++ module

  Code Block
  bgColor #F7F7ED
  ImgAlgos/app/corana.cpp (or ImgAlgos/test/corana.cpp)

  Note
  Note, the application in the test directory is compiled and run by the commands: scons test <path>/corana -f <fname_data> -t <fname_tau>\ -h -l <logfile>\ -b <basedir>\

  where

• <fname_data> is one of the data files: <fname_prefix>-<fname-common>-b<block-number>.<file_type>, which needs to be available;
• <fname_tau> is a file with a list of indexes of tau for evaluation of correlations. By default or if the file is missing, the list of indexes will be generated automatically, and for book-keeping is saved in the file <fname_prefix>-<fname-common>-tau.txt;
• <basedir> is a directory for all data files, which is current by default;
• <logfile> is an output log-file, or standard output by default.

Module ImgAlgos::ImgTimeStampList

This module is reduced from ImgAlgos::ImgVsTimeSplitInFiles.
Functionality is restricted to:

• saves counter number and the time records in file fname.

• print summary parameters for parser, for example:

  Code Block
  bgColor #F7F7ED
  ImgTimeStampList: Summary for parser BATCH_RUN_NUMBER 0020 BATCH_NUMBER_OF_EVENTS 75 BATCH_FRAME_TIME_INTERVAL_AVE 8.086934 BATCH_FRAME_TIME_INTERVAL_RMS 0.120381 BATCH_FRAME_TIME_INDEX_MAX 74

Module ImgAlgos::UsdUsbEncoderFilter

ImgAlgos::UsdUsbEncoderFilter (ImgAlgos > V00-03-43) module is created by request of Thomas Kroll for experiment with mobile rack in SACLA .

This psana module contains an example of how to get UsdUsb::DataV1 object and time-stamps from different data sources for event synchronization purpose.

Functionality:

To work with time-code objects it uses helper class TimeCode. It loads input timing information from file defined by the  ifname  and store it in the std::vector<TimeCode> member object. For each event the TimeCode object is defined from data; time stamp from PSEvt::EventId, and unique code from UsdUsb::DataV1. TheTimeCode object from data is compared with information loaded from the input file. If the TimeCode object from data is consistent with one of the records in the file event is passed for further processing, otherwise discarded.

Input (ifname) and output (ofname) files have any number of records of the same format. Each record consists of four integer numbers:

•     timestamp sec (uint32_t),
•     timestamp nsec (uint32_t),
•     UsdUsb 6-bit code (uint8_t),
•     user defined counter (unsigned).

Example of the input/output file content:

1373280273 293197261  51     1
1373280273 301532011  52     2
1373280273 309867156  54     3
1373280273 318206221  58     4
1373280273 326553079  61     5
...

See also Example for Module ImgAlgos::UsdUsbEncoderFilter.

Module ImgAlgos::ImgIntMonCorr

This module is intended for CorAna project.
ImgIntMonCorr gets the image and intensity monitor data, evaluate the normalization factor, applies this factor to the image intensity, and saves the corrected image in the event.

Functionality:

• ImgIntMonCorr uses the source and key_in parameters to get the input image (as ndarray<const double,2> object),
• gets and process the intensity monitors' data in accordance with configuration from file fname_imon_cfg,
• intensity normalized image is saved in the event with key key_out. The type of output data is the same as the type of input data.

fname_imon_cfg file content per line: source name, short name, on/off bits for 4-channels, normalization and selection, minimal, maximal and averaged intensities:

 BldInfo(FEEGasDetEnergy)         FEEGasDetEnergy  1 1 1 1  0 1   3.000000  7.500000  5.250000
 BldInfo(XCS-IPM-02)              XCS-IPM-02       1 1 1 0  1 0  10.000000 13.000000 11.500000
 BldInfo(XCS-IPM-mono)            XCS-IPM-mono     1 1 1 0  0 1  14.500000 16.000000 15.250000
 DetInfo(XcsBeamline.1:Ipimb.4)   Ipimb.4          1 1 1 1  0 0  -1.000000 -1.000000  1.000000
 DetInfo(XcsBeamline.1:Ipimb.5)   Ipimb.5          1 1 1 1  0 0  -1.000000 -1.000000  1.000000

Module ImgAlgos::IntensityMonitorsData

This module is intended for CorAna project.
It gets the 5 intensity monitors data (4 channels for each) and saves them in the text or binary file file_data. Comments (or header) for this file is saved separately in file_header. It also prints the summary parameters for parser, for example:

IntensityMonitorsData: Summary for parser
BATCH_RUN_NUMBER              0020
BATCH_NUMBER_OF_EVENTS        75

Module ImgAlgos::CSPadArrSaveInFile

This module saves the CSPad data array formatted as [5920=4*8*185][388] in output file for each passes event.

Module ImgAlgos::CSPadArrAverage

This module averages the CSPad data array and saves two files for averaged and rms values in CSPad format [5920=4*8*185][388]. In contrast to the cspad_mod.CsPadPedestals, the input data can be specified with a key, that allows to average CSPad array for already pre-processed data, for example "calibrated". This feature can be used to evaluate the averaged signal or background event. Implemented algorithm of averaging allows to eliminate large statistical fluctuations in the pixel amplitude spectrum. In advanced case averaging may have up to three stages, depending on configuration parameters:

• 0-stage: the 1st portion of events from 0 to evts_stage1 is averaged without any constrains, the preliminary averaged and rms values are defined for each pixel
  at the end of this stage.
• 1-stage: starting from the event evts_stage1 data are collected only for abs(amplitude-average0) < gate_width1. At the end of this stage the preliminary averaged and rms values are defined for each pixel.
• 2-stage: starting from the event evts_stage1 + evts_stage2 data are collected only for abs(amplitude-average1) < gate_width2. At the end of this stage the preliminary averaged and rms values are defined for each pixel and saved in the files specified by the avefile and rmsfile parameters, respectively.

This type of averaging algorithm may be useful for pedestal defenition in case of large amplitude fluctuations.

Default version of the configuration parameters works the same way as cspad_mod.CsPadPedestals. In this case module gets raw events and stage 0 continues for entire input data sample.

See also Example for Module ImgAlgos::CSPadArrAverage.

Module ImgAlgos::CSPadCommonModeCorrection

Alternative to the cspad_mod.CsPadCalib algorithm for the common mode correction.
Takes the CSPad data array with subtracted pedestals, evaluate the average amplitude for each 2x1 section for amplitudes below the threshold, and subtract it from all pixel amplitudes. This algorithm consumes about 30 ms/event on psana0205.

Module ImgAlgos::CSPadBkgdSubtract

This module uses the CSPad array, specified by the configuration parameters source and inputKey, subtracts the background, defined in the file bkgd_fname, and saves the resulting array in the event with outputKey. The subtracted background array is normalized on the sum of pixel amplitudes in the quad section norm_sector, which can be set from 0 to 7.

   shared_ptr<Psana::CsPad::ConfigV2> config2 = env.configStore().get(m_str_src);
   unsigned mask = config2->roiMask(quad_number); // should be in the range from 0 to 255

See also Example for Module ImgAlgos::CSPadBkgdSubtract.

Module ImgAlgos::CSPadMaskApply

This module uses the CSPad array, defined by the configuration parameters source and inkey, apply the mask from file mask_fnname and saves the masked data with key outkey. For masked pixels the amplitude will be replaced by the value from masked_amp.

The file mask_fnname has the same structure as files for pedestals and background with dimensions [4*8*185][388]. Masked pixels are indicated by 0-th in this file. This file can be generated, for example, from the averaged background file, using amplitude threshold. This can be done with auxiliary python script MakePixelMask.py as explained in Example for Module ImgAlgos::CSPadMaskApply.

Module ImgAlgos::CSPadArrNoise

This module works on CSPad data array shaped as [5920=4*8*185][388], uses the "median algorithm" to evaluate the signal and noise for each pixel, evaluates S/N ratio for each pixel, counts the fraction of events where S/N > SoNThr, and writes the same shape arrays for pixel mask and status information in the maskfile and statusfile, respectively. The statusfile contains for each pixel the fraction of events where S/N > SoNThr. This module presents a part of features implemented in the module ImgAlgos::CSPadArrPeakFinder.

See also Example for Module ImgAlgos::CSPadArrNoise.

Module ImgAlgos::CSPadArrPeakFinder

Module ImgAlgos::CSPadArrPeakFinder is a psana-based implementation of the "median algorithm" for peak finding in CSPad data array shaped as [5920=4*8*185][388]. This algorithm was first implemented in myana/Cheetah by Anton Barty and Co. The "median algorithm" assumes that the amplitude level of background and noise for each pixel can be estimated as a mean and RMS of the surrounding pixels, located in the ring with parameters rmin and dr around the pixel in question. The threshold SoNThr_noise on signal over noise (S/N) ratio allow to asset the pixel amplitude as a large noise fluctuation. Statistics of pixels above the S/N threshold accumulated over many images can be used to form the noisy-pixel mask. For example, if the fraction of images where pixel exceeds the S/N threshold grater than certain value (frac_noisy_imgs=0.9), the pixel is considered as noisy. The permanent bad pixel mask (see module ImgAlgos::CSPadMaskApply) and dynamically evaluated noisy pixel mask are used to get rid of bad pixels and improve the image quality. Healthy pixels with S/N above threshold (SoNThr_signal about 3-5) are treated as potential signals. Using recursive flood-filling algorithm the groups of connected signal pixels can be found and considered as a candidate for a diffraction peaks. Peak finding algorithm uses the amplitude, S/N thresholds, and limits on number of pixels in the connected region (parameters peak_amp_tot_thr, peak_SoN_thr, peak_npix_min, and peak_npix_max) in order to define the peak. Finally, the event is selected or discarded depending on number of found peaks and total amplitude threshold, defined by the parameters event_npeak_min, event_npeak_max, and event_amp_tot_thr, respectively.

Description of implemented algorithm:

• in the constructor and beginJob(...)method:
  • enter input parameters,
  • (re)set the initial mask of noisy pixels from file hot_pix_mask_file (if its name is specified in the configuration file),
  • do necessary initialization of work arrays.
• in the event(...)method the main part of "median algorithm" is implemented:
  • fill [4][8][185][388] per-pixel arrays:
    • m_stat - number of events with |S/N| > SoNThr,
    • m_signal - signal amplitude, or 0(zero) for masked pixels,
    • m_proc_status - sets 255 for S/N > SoNThr or 0(zero) for masked pixels.
  • use arrays m_proc_status and m_signalto find peaks:
    • iterate over [185][388] 2x1 pixels and find the connected regions (using recursive flood-filling algorithm)
    • create vector of peaks v_peaks of struct Peak, using peak_npix_min, peak_npix_max, and peak_amp_tot_thr parameters,
  • loop over v_peaks, count total amplitude and the number of peaks in the event.
  • decide if the event selected or not based on event_npeak_min, event_amp_tot_thr, and selection_mode parameters.
  • periodically dynamically re-generate the mask, based on m_stat array and frac_noisy_imgs parameter. When to start and for how many events to update the mask is defined by the nevents_mask_update and nevents_mask_accum parameters, respectively.
  • save m_signal in file for selected events, depending on out_file_bits parameter.
  • put the vector with peaks v_peaks in the evt with key=key_peaks_out.
• in the endJob(...) method, depending on bit status in out_file_bits:
  • save current hot-pixel mask in the file hot_pix_mask_out_file
  • save current fraction of events with noisy/signal pixels in the file frac_noisy_evts_file

See also Example for Module ImgAlgos::CSPadArrPeakFinder.

Module ImgAlgos::CSPadArrPeakAnalysis

This module is intended for analysis of the results obtained in the peak finding algorithm implemented in the ImgAlgos::CSPadArrPeakFinder module.

1. It gets the vector of peaks defined by the key parameter and prints it.
2. fills ROOT-style histograms and ntuples, and
3. saves histograms and ntuples in file defined by the fname_root parameter - NOT AVAILABLE SINCE V00-03-50 - get rid of root...

See also Example for Module ImgAlgos::CSPadArrPeakAnalysis.

Package pyimgalgos

This package contains python modules which work with both frameworks pyana and psana. Functionality of these modules resembles modules from C++ package ImgAlgos. The difference between two frameworks at code level is explained in Migration from pyana to psana.

Module pyimgalgos.cspad_arr_producer

This module gets data from evt store for CSAPD or CSPAD2x2 depending on unique detector name in parameter source, produce numpy array of full scale shape (4,8,185,388) or (185, 388, 2) of specified in dtype type, and saves it in the evt store with unique name key_out. In case of missing 2x1 sections, their pixel amplitudes substituted by the value form val_miss.

See also Examples for package pyimgalgos.

Module pyimgalgos.cspad_image_producer

This module gets from evt store the numpy array identified by key_in of full scale shape (4,8,185,388) or (185, 388, 2) for CSPAD or CSPAD2x2, respectively, and produces 2-d image numpy array, taking into account geometry calibration parameters specified by the path calib_dir. The output image array is saved in the evt store with unique name, specified by parameter key_out.

See also Examples for package pyimgalgos.

Module pyimgalgos.image_crop

This module gets from evt store the 2-d image numpy array identified by source and key_in, crop it using range of row and column parameters, and saves cropped 2-d image numpy array  in the evt store with unique key_out.

Module pyimgalgos.image_save_in_file

This module gets from evt store 2-d image numpy array for specified unique name in key_in and saves it in the file with name given by parameter ofname. File extension defines the output file format. Experiment, run, and event numbers are added to the name of the output file. For example, for ofname = image.tiff files will be created with names image-<experinent>-r####-ev######.tiff, where symbols # stands for numbers.

In the event browser mode, mode>0, this module saves images in the files with names from the ring buffer. That images can be seen by the application plims, see command options:

% plims -h

See also Examples for package pyimgalgos.

Module pyimgalgos.tahometer

Is intended to print records showing job performance current and integrated from the beginning of job:

pyimgalgos.tahometer: run:0049  evt:000005  t[sec]:     2.575  dt[sec]:     2.575  n/t[1/sec]:     1.942  dn/dt[1/sec]:     1.942

See also Examples for package pyimgalgos.

Module pyimgalgos.ex_peaks_nda

This module is an example of how to get from the evt store the 2-d  numpy array of shape=[Npeaks,12] with peak info produced by ImgAlgos.CSPadArrPeakFinder.

The numpy array is identified by source and key_in. If the peak numpy array is available in the event it will be printed.

See also Example for Module ImgAlgos::CSPadArrPeakAnalysis.

Package Translator

The translator package include the H5Output module which translates xtc to hdf5. For more  information see the page The XTC-to-HDF5 Translator

Package psana_test

Anchor
psana_test psana_test

The psana_test package includes the psana module dump. This will take a standard psana datasource and dump all the event, config, and epics data found. The entire contents of large arrays are not printed. However a checksum over all the array data is, as well as the min, 25th percentile, median, 75th percentile, and max over the data. The dump module does not serve as a good example of how to retrieve and work with objects from the event store – see the psana_examples package for this. The psana_test package is primarily for psana developers to do software testing.

module dump

Running psana_test dump

An example of running the module is

psana -n 2 -m psana_test.dump exp=xpptut13:run=179

This dumps the first two events of run 179 of the xpp tutorial data.

Understanding psana_test dump output

Below we annotate the output that psana_test.dump can produce. All annotations are preceded by a #

==================
=== begin job ====
# first the epics aliases are printed during begin job.
Epics Aliases: total = 240
  Be_xpos
  Be_ypos
  Be_zpos
  ... 
# next the epics pv, as they appear during beginJob.
# This corresponds to the xtc configure transition. 
# At this point, these are ctrl pvs.
Epics PV
  pvName=HX2:DVD:GCC:01:PMON  pvid=106 dbrtype=34 isCtrl=1 pvName=HX2:DVD:GCC:01:PMON numElements=1 status=0 severity=0 units=T upper_disp_limit=1.0000e-02 lower_disp_limit=0.0000e+00 upper_alarm_limit=0.0000e+00 upper_warning_limit=0.0000e+00 lower_warning_limit=0.0000e+00 lower_alarm_limit=0.0000e+00 upper_ctrl_limit=1.0000e-02 lower_ctrl_limit=0.0000e+00 data=5.0000e-09
  ...
# After epics, we get the content of the psana env configStore. 
# The dump module is getting all keys from the configStore(), 
# then retrieving each object. Objects that have a xtc type id, or are
# an numpy array will be printed.

# For each object, we first get a string describing the even key:

type=psana.ControlData.ConfigV2, src=ProcInfo(0.0.0.0, pid=7670)

  # then we get the data of the object.
  # most all data for a Psana object is obtained through accessor methods.
  # methods that return unsigned ints print in hex. 
  # methods that return signed ints print in decimal.
  # methods that return floats print in scientific format with 4 decimals of precision.

  npvControls: 0x1      
  npvMonitors: 0x0    
  npvLabels: 0x0
  events: 0x1E0
  uses_duration: 0x0
  uses_events: 0x1

# methods that return a compound type, like duration() that returns the compound type time
# are printed as follows:
  duration:
    nanoseconds: 0x0
    seconds: 0x0

# some methods return a python list. Each element in the list is printed separately:
  pvControls[0]:
    name: las_lensh
    index: 0xFFFFFFFF
    value: 0.0000e+00
    array: 0x0
  ...

# epics config is not dumped, see the EPICS alias list for how Psana stores this data
type=psana.Epics.ConfigV1, src=DetInfo(EpicsArch.0:NoDevice.0)
  epicsConfig not dumped

# cspad config is an example which has methods that return ndarrays
type=psana.CsPad.ConfigV4, src=DetInfo(XppGon.0:Cspad.0)
  ...
  numSect: 0x20
  # some methods return a list of simple types, these are printed in one line
  roiMask [0]=0xFF [1]=0xFF [2]=0xFF [3]=0xFF
  numAsicsStored [0]=0x10 [1]=0x10 [2]=0x10 [3]=0x10
  ...
  quads[0]:
    ...    
    dp:
      # for an ndarray, we print the type, dimensions, adler32 checksum, and quartile
      # statistics (min, 25th percentile, median, 75th percentile, and max):
      pots: ndarray_uint8_1: dim=[ 80 ] adler32=0x231B31F5 min=0x0 25th=0x3F median=0xB0 75th=0xFF max=0xFF
    gm:
      gainMap: ndarray_uint16_2: dim=[ 185 x 194 ] adler32=0x18730001 min=0x0 25th=0x0 median=0x0 75th=0x0 max=0x0
   ...

===============================================================
=== beginrun 0 ===            # typically, there is nothing new in beginrun
===============================================================
=== begincalibcycle run=0 step=0 ===

# having dumped the entire initial contents of epics and the config store, 
# the dump module will now only print changes to epics or the config.
# That is it remembers how each epics pv and config object printed the last 
# time it saw it. With each new transition, it looks at all the epics pv and
# config objects. If any change, they are dumped.

# The control data changed in the calib cycle:

type=psana.ControlData.ConfigV2, src=ProcInfo(0.0.0.0, pid=7670)
  npvControls: 0x1
  npvMonitors: 0x0
  npvLabels: 0x0
  events: 0x1E0
  uses_duration: 0x0
  uses_events: 0x1
  duration:
    nanoseconds: 0x0
    seconds: 0x0
  pvControls[0]:
    name: las_lensh
    index: 0xFFFFFFFF
    value: -4.9997e-01
    array: 0x0

# next we see event data, printing the following:
===============================================================
=== event: run=0 step=0 event=0 seconds= 1362889345 nanoseconds= 770371931 fiducials= 19593
# at this point, all epics pv's are replaced with TIME pv's, not the stamp.sec, stamp.nsec below:
Epics PV
  pvName=HX2:DVD:GCC:01:PMON  pvid=106 dbrtype=20 isTime=1 numElements=1 status=0 severity=0 stamp.sec=731737344 stamp.nsec=134374000 data=5.2000e-09
  ...
# while most epics pv's have one value, there are some with more than one. If a EPICS pv has less than 20 values, they are all printed, otherwise the typical ndarray summary is printed.

# now we get into regular event data
type=psana.EvrData.DataV3, src=DetInfo(NoDetector.0:Evr.0)
  numFifoEvents: 0x2
  fifoEvents[0]:
    timestampHigh: 0x4C89
    timestampLow: 0x32A6
    eventCode: 0x29
  fifoEvents[1]:
    timestampHigh: 0x4C89
    timestampLow: 0x2E4C
    eventCode: 0x8C
type=psana.CsPad.DataV2, src=DetInfo(XppGon.0:Cspad.0)
  quads[0]:
    seq_count: 0x1
    ticks: 0x329D
    fiducials: 0x4C89
    sb_temp: ndarray_uint16_1: dim=[ 4 ] adler32=0x4FC00AC min=0x7 25th=0x8 median=0x9 75th=0x291 max=0x291
    frame_type: 0x4
    data: ndarray_int16_3: dim=[ 8 x 185 x 388 ] adler32=0xAD5ACF7F min=0 25th=1281 median=1346 75th=1475 max=16383
    virtual_channel: 0x0
    lane: 0x0
    tid: 0x0
    acq_count: 0x85
    op_code: 0x85
    quad: 0x0
    sectionMask: 0xFF
  quads[1]:
    seq_count: 0x1
    ticks: 0x329D
    fiducials: 0x4C89
    sb_temp: ndarray_uint16_1: dim=[ 4 ] adler32=0x2DB006A min=0x7 25th=0xA median=0xA 75th=0x34B max=0x34B
    frame_type: 0x4
    data: ndarray_int16_3: dim=[ 8 x 185 x 388 ] adler32=0xD28441BE min=0 25th=1316 median=1374 75th=1504 max=16383
    virtual_channel: 0x0
    lane: 0x0
    tid: 0x0
    acq_count: 0x85
    op_code: 0x85
    quad: 0x1
    sectionMask: 0xFF

src aliases

If a source alias has been defined, it will show up when the event key is printed:

type=psana.Pimax.FrameV1, src=DetInfo(AmoEndstation.0:Pimax.0) alias=pimax
 

Options

Several options allow you to control the output of psana_test.dump

The most useful are

include = term1 term2
exclude = term1 term2

These are used to filter the key strings. For example, running

psana -m psana_test.dump -o psana_test.dump.include=10k exp=mob30114:run=145

Would only dump event keys that had 10k in them, effectively giving you only Epix::Config10KV1 and psana.Epix.ElementV1 since these are the only types coming from the source DetInfo(NoDetector.0:Epix10k.0).

Other options one could set are:

epics = False       do not print epics
aliases = False     do not print the EPICS alias list
dump_aliases=True   follow EPICS aliases to print the EPICS pv's they point to
regress_dump=True   do not print the DAQ assigned pvId when printing EPICS
config = False      do not print the contents of the configStore, only regular event data
counter = False     do not print the counter string that labels event numbers and calib cycle numbers
indent = 4          change the indent from the default of 2 to 4

Library Usage

Two functions are provided in the Python psana_test package that allow Python scripts to turn Psana objects into strings. A Python script could include the following function to build dictionaries describing the state of the event, configStore, and epicsStore:

from psana_test import obj2str, epicsPvToStr

def getPsanaState(event, configStore, epicsStore):
    evtDict = {}
    cfgDict = {}
    epicsDict = {}
    for key in event.keys():
        if key.type() is None: continue
        obj = event.get(key.type(), key.src(), key.key())
        if (obj is None): continue
        if not hasattr(obj,'TypeId'):  continue
        evtDict[str(key)]=obj2str(obj)
    for key in configStore.keys():
        if key.type() is None: continue
        obj = configStore.get(key.type(), key.src())
        if (obj is None): continue
        if not hasattr(obj,'TypeId'):  continue
        cfgDict[str(key)]=obj2str(obj)
    for pvName in epicsStore.pvNames():
        pv = epicsStore.getPV(pvName)
        if not pv: continue
        epicsDict[pvName] = epicsPvToStr(pv)
    return evtDict, cfgDict, epicsDict

Package TimeTool

Modules for analyzing recorded data from a timetool camera setup. The timetool camera measures the time difference between laser and FEL in one of two methods:

1. spatial encoding, where the X-rays change the reflectivity of a material and the laser probesthat change by the incident angle of its wavefront; or
2. spectral encoding, where the X-rays change the transmission of a material and the chirped laser probes it by a change in the spectral components of the transmitted laser.

Below the package modules are described. The package includes sample configuration files that describe all the options. From a psana release directory, users are encouraged to add the TimeTool package to obtain the latest source. For instance:

newrel ana-current myrel
cd myrel
kinit             # get ticket to check out package from svn repository
addpkg TimeTool
sit_setup
scons
# now examine the files in TimeTool/data:  sxr_timetool.cfg  sxr_timetool_setup.cfg  timetool_setup.py  xpp_timetool.cfg  xpptut.cfg

 timetool_setup.py is a python script to calculate the digital filter weights.

Module Analyze

a module that analyzes the camera image by projecting a region of interest onto an axis and dividing by a reference projection acquired without the FEL.  The resulting projection is processed by a digital filter which yields a peak at the location of the change in reflectivity/transmission.  The resulting parameters are written into the psana event. The default behavior is to write these values in as a set of double's. Unfortunately double's are presently not visible to Python scripts or modules in the psana framework. In the next analysis release (ana-0.13.1) these values can also written as a set of ndarray's, each with one double. These will be visible on the Python side. This can be done by adding the option

put_ndarrays=True

to the config file. To use the features prior to that release, when adding the TimeTool package to your test release, do

addpkg TimeTool V00-00-03

Module Check

a module that retrieves results from the event for either the above module or from data recorded online.

Module Setup

a module that calculates the reference autocorrelation function from events without FEL for use in the digital filter construction.

References

• Psana User Manual - Old
• Psana Reference Manual - Old
• psana - Module Examples
• Migration from pyana to psana
• psana - Migration from pyana

Functionality

For data source  in each run loads/updates calibration geometry file from the calibration DB, evaluates pixel coordinate arrays using class PSCalib::GeometryAccess and saves them as ndarray<const float,1> in the event store for keywords x-pix-coords, y-pix-coords, and z-pix-coords.

The main idea of this module is that calibration geometry file will be found and loaded (if available) automatically.

Configuration parameters

Current version of this module works with CSPAD and CSPAD2x2. It can be extended for other detectors, whenever necessary.

See Example for Module ImgAlgos::PixCoordsProducer,