Content

Common mode correction

Most of pixel array detectors produce imaging data that can not directly be used in analysis and need in corrections. Most popular corrections are

• dark rate (pedestal) subtraction,
• bad pixel masking,
• common mode correction,
• gain correction,
• etc.

In this note we discuss common mode correction algorithms. Common mode is a hardware effect of collective pixel intensity variation due to synchronous variation of potentials on sensor chip or ADC at readout process. This effect can be corrected at low sensor illumination, where the number of pixels with energy deposition from photons is small. The spectrum of pixel intensities without photons should be grouped in the peak with small offset of the average from zero (if the dark rate correction is already applied) due to the common mode effect. This offset can be evaluated and subtracted from all pixel intensities in the group of synchronously fluctuating pixels. The number and structure of commonly behaving pixel groups depend on detector hardware.

Implementation

Below we describe common mode correction algorithms used in

• psana module ImgAlgos::NDArrCalib. This correction along with others can be applied to raw ndarray under control of the psana configuration file for specified detector, for example for  Epix100a:

[ImgAlgos.NDArrCalib]
source  = DetInfo(:Epix100a)
key_in  = ndarray-raw
key_out = ndarray-clb
do_peds = yes
do_cmod = yes
do_stat = yes
...

where keywords do_peds and do_cmod should have value yes or true in order to turn on pedestal subtraction and common mode correction algorithms.

• Detector.PyDetector package which provides uniform access to calibration and data for all detectors.

Low level implementation of the common mode correction algorithms is done in class ImgAlgos::CommonModeCorrection which uses methods from ImgAlgos::CommonMode.

Each algorithm may load file with parameters from calibration directory, which by default accounts for experimnet, calibration version, data source, calibration type and run range:

/reg/d/psdm/<INS>/<experiment>/calib/<calib-version>/<data-source>/<calibration-type>/<run-range>.data

For example:
/reg/d/psdm/XPP/xppi0614/calib/Epix100a::CalibV1/NoDetector.0:Epix100a.0/pedestals/0-end.data
/reg/d/psdm/XPP/xppi0614/calib/Epix100a::CalibV1/NoDetector.0:Epix100a.0/common_mode/0-end.data

Content of this file depends on detector, calibration type, and algorithm, as shown below.

Algorithms

We use algorithms earlier developed for CSPAD and other detectors and currently residing in ImgAlgos and  psalg packages.

Selection of algorithm of particular type is controlled by the parameters in file for common_mode calibration type.

#1 - common mode peak finding algorithm

Valid for: CsPad and CsPad2x2

This algorithm is similar to one developed by Andy Salnikov and implemented in cspad_mod.CsPadCalib;

1. for each cspad2x1 sensor the pixels' intensity histogram is filled for natural ADU bins, pixels with bad status are ignored,
2. search for the peak in the histogram corresponding to the dark pixels using thresholds,
3. iterate over bins around the peak region and improve the peak location precision, that defines the common mode correction
4. if correction is in allowed range then apply it to all sensor pixels.

Control parameters for this algorithm resides in the file for common_mode calibration type;

 parameters for CSPAD and CSPAD2x2

1 50 50 100

• par[0] - algorithm #
• par[1] - maximal deviation of the peak mean from 0
• par[2] - maximal allowed value of the peak RMS
• par[3] -threshold on number of pixels in the ADU bin in the peak finding algorithm

parameters for other detectors

• par[4] - number of segments for common mode evaluation
• par[5] - segment size (number of pixels for common mode evaluation)

• par[6] - stride (step for jump to the next pixel)

  For example:

  1 50 50 100 8192 128 1

#2 - MEAN algorithm

Not valid for: CsPad and CsPad2x2

It was developed by Philip Hart for test purposes;

1. for each group of pixels intensity histogram is filled for natural ADU bins, pixels with bad status are ignored,
2. a simple mean below threshold is considered as a common mode correction,
3. if correction is in the allowed range, it is applied to all pixels from the group.

Control parameters for this algorithm resides in the file for common_mode calibration type;

• par[0] - algorithm #
• par[1] - maximal threshold on intensity to evaluate mean for low intensities
• par[2] - maximal allowed common mode correction

• par[3] - length of consecutive pixel array for common mode evaluation
  For example, for pnCCD one can evaluate common-mode for one came chip (128 channels).

  Control file:   /reg/d/psdm/amo/<exp-name>/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/calib/PNCCD::CalibV1/Camp.0:pnCCD.0/common_mode/0-end.data

  2 1000 1000 128

• The algorithm loops over the data and evaluates consecutive arrays of specified length (which might represent for example a row of pixels in a readout chip) and finds the mean value for values below a threshold, ignoring masked pixels.  It corrects all the data if the calculated common mode is less than the maximal allowed correction.  The median algorithm is more stable in many cases and is recommended.

#3 - MEDIAN algorithm

Not valid for: CsPad and CsPad2x2

It was developed by Philip Hart as a replacement for the mean (#2) algorithm;

1. for each group of pixels intensity histogram is filled for natural ADU bins, pixels with bad status are ignored,
2. a half of statistics counted from bin with low intensities below threshold is considered as a common mode correction,
3. if correction is in the allowed range, it is applied to all pixels from the group.

Parameters are the same as in #2.  The algorithm is as above, except that it calculates the median, or the average of the two median points if there are an even number passing the selection criteria.

3 100 100 128

#4 - MEDIAN algorithm - detector dependent

Not valid for: CsPad and CsPad2x2

It is pretty similar to one developed by Matthew Weaver that is implemented in ami/event/FrameCalib

In ImgAlgos::NDArrCalib it is implemented for Epix100a and  Fccd960.  The algorithm is detector-dependent, executing different code depending on whether an Epix or Fccd "Source" is given to the NDArrCalib module.

The difference from algorithm #3 is quite minor; it starts to fill the intensity histogram in a quite narrow range relative to zero, but if the half of statistics is not found in this range it is extended by 1/4 of pixels (see parameter #3 below for initial guess for range of histogram). This iterations are repeated until the half of statistics is in the range, or the number of bins exceeds 10000. 

Typical common_mode  control parameters:

4 6 30 10    0 0 0 0    0 0 0 0    0 0 0 0    0 0 0 0

where

1. par[0] (4) - algorithm number; 4 stands for this median algorithm,
2. par[1] (1) - type of regions for median algorithm; 3 regions (banks/rows/columns) for Epix100a, and 2 regions (banks/rows) for Fccd960:
   

   1. EPIX100A has an option to turn on up to 3 regions for common mode correction, controlled by the bitword parameter #2: bit#1 - common mode for 352x96-pixel 16 banks,  bit#2 - common mode for 96-pixel rows in 16 banks,  bit#3 - common mode for 352-pixel columns in 16 banks

   2. FCCD960 has 2 regions, also selected by parameter #2:  bit#1 - common mode correction for 1x160-pixel rows with stride 2, bit#2 - common mode correction for 480x10-pixel 96*2 supercolumns

3. par[2] (30) - initial guess for range of histogram (relative to 0) in ADU (see above description of intensity histogram).  This number can grow in subsequent iterations if half of the statistics is not found in this range.  It should probably start at around 3 sigma of the noise value.
4. par[3] (10) - maximal allowed absolute value of the common mode correction in ADU. If = 0 - no-limit is applied directly, indirect limit = 10000ADU (from limit on intensity histogram described above).

Other parameters are not used.

#5 - Unbond pixels common mode correction

Valid for CsPad and CsPad2x2 only!

In latest version of CSPAD (including 2x2) detectors a group of pixels is not bound to relevant electronic channels. For each 2x1 averaged signal from unbound pixel-channels is used as a common mode variation value. It is subtracted from all other pixels.

Typical common_mode  control parameters:

5 50

where

• par[0] - algorithm #,
• par[1] - maximal allowed correction value.

#6 - Epix neighbor exclusion

The mask created by this algorithm includes, along with the pixels above the threshold, their neighbors. The statistics will be poorer because there will be less pixels used in the common mode, but possible leakage from the pixels containing photons will not be included. The parameters are:

1. img - image on which the common mode is applied
2. rms - array of noise values for each pixel with same shape as img

3. maxCorr - (default = 30) maximum correction applied. If common mode correction is larger than this value, no correction will be applied

4. histoRange - (default = 30) all pixels above this parameter are masked

5. colrow - (default = 3) decides what is corrected. If 1, only the columns are corrected. If 2, only the rows are corrected. And if 3, both are corrected

6. minFrac - (default = 0.25) the minimum fraction of pixels required to be left in a row or column after applying the mask and rejecting high pixels and their neighbors

7. normAll - (default = False) if true, will subtract the mean from the full image with the masked applied

Below is an example block of code showing how to call this algorithm and pass it arguments for some dataset and detector.

for nevent, evt in enumerate(ds.events()):
	if nevent == 10:
		break
 
	# cmpars = [6] because this is the 6th common mode algorithm and it must be in a list format
	# Add arguments on as seen fit
    nda = det.calib(evt, cmpars = [6], rms = det.rms(evt), maxCorr = 25)

Test of the common mode correction for pnCCD

To test implementation of algorithms in ImgAlgos::NDArrCalib we use the same data sets as in 2014-03-25-Ankush-CommonModeNoise.pdf

Use data from experiment amob5114

High gain pnCCD run 121

2(or 3) 1000 1000 128

Spectra for 1) raw data, 2) subtracted pedestals, 3) subtracted common mode correction algorithm #2 and 4) algorithm #3:

Images 1) for subtracted pedestals and 2) common mode correction algorithm #2:

High gain pnCCD run 329

2(or 3) 1000 1000 128

Spectra for 1) raw data, 2) subtracted pedestals, 3) subtracted common mode correction algorithm #2 and 4) algorithm #3:

Images 1) for subtracted pedestals and 2) common mode correction algorithm #2:

Summary for pnCCD

Common mode correction for pnCCD

• gives significant effect in low gain mode and is negligible in high gain mode
• algorithm #2 gives the best results, #3 a little bit worse, #1 - does not work for pnCCD

Test of common mode correction for CSPAD

Use cxi83714-r0136 with

1 10 10 100

Spectra for 1) raw data, 2) subtracted pedestals, 3) subtracted common mode correction algorithm #1:

Images 1) for subtracted pedestals and 2) common mode correction algorithm #1:

Comparison of Unbonded vs. Default Common Mode

Dark Case

This is for a dark run, and was generated using this script:

import psana
from matplotlib import pyplot as plt
import numpy as np
dataset_name = "exp=cxil5316:run=1"
ds = psana.DataSource(dataset_name)
psana_det = psana.Detector("DscCsPad")
evt = ds.events().next()
pedestal = psana_det.pedestals(evt)
data = psana_det.raw_data(evt) - pedestal
data_cm = psana_det.raw_data(evt) - pedestal
unbond_cm = psana_det.common_mode_correction(evt, data_cm, [5, 50])
default_cm = psana_det.common_mode_correction(evt, data_cm, [1, 50, 10, 100])
unbond=unbond_cm[:,0,0]
default=default_cm[:,0,0]
plt.subplot(2,1,1)
plt.title('Default vs. Unbond Correction (ADU)')
plt.plot(unbond,default,'o')
plt.subplot(2,1,2)
plt.title('Default-Unbond Correction (ADU)')
plt.hist(default-unbond)
plt.show()

This yields the following plots comparing the size of the two corrections:

Brighter Case

This is the same plot for an event with many photons.

dataset_name = "exp=cxid9114:run=96:idx"
if t.time() == 6025277111415285948L:

The common-mode methods still correlate, but the default method perhaps over-subtracts because of leakage into the zero-photon peak from neighbor-pixels of real photons.

Crystallography Case

Here are the common mode corrections applied to a crystal diffraction pattern with a strong water ring using algorithms 1 vs 5. Algorithm 5 seems to perform better in this case.

Common mode histogram using algorithm 1

Common mode histogram using algorithm 5

Thoughts from Phil on CsPad Common Mode:

• statistics of unbonded pixels is a problem
• poisson statistics on bragg spots is bigger than the common mode
• don't correct common-mode for low-gain pixels
• the larger shifts seen in bright events are "cross-talk" caused by the "ramp" where pixels nearer the mean-pedestal have a lower gain
• should do per-asic common-mode
• should do algorithm 5, then algorithm 1 if it fails

Summary for CSPAD

Common mode correction for CSPAD works with algorithm #1 and shows significant effect.

Test of common mode correction for CSPAD2x2

Use meca1113-r0045 with

1 50 10 (100) - last parameter is set by default

Spectra for 1) raw data, 2) subtracted pedestals, 3) subtracted common mode correction algorithm #1:

Images 1) raw data and 2) subtracted pedestals with common mode correction algorithm #1:

Summary for CSPAD2x2

Common mode correction for CSPAD2x2 in this example shows minor improvement.

References

2014-03-25-Ankush-CommonModeNoise.pdf - stand-alone test of common mode correction for pnCCD

psana - Module Catalog - Module ImgAlgos::NDArrCalib

Epix100a

Fccd960-Detector