...

Dark data processing algorithms include pixel status evaluation which is saved in calibration files of type pixel_status. This note explains how to find definition and statistics of the bad pixel bits.

Algorithm

...

Dark data processing algorithm

Dark data processing scripts dump in the log file information about bad pixel statistics, for example in case of jungfrau:

raw data found/selected in 999 events
[I] L0688 begin data summary stage
[I] L0111 evaluate_limits RMS: ave=4.783 std=1.303 limits low=0.001 high=12.604
[I] L0111 evaluate_limits AVE: ave=14191.873 std=457.903 limits low=11444.455 high=16000.000

[I] L0733 bad pixel status:
status 1: 244 pixel rms > 12.604
status 2: 550 pixel rms < 0.001
status 4: 418 pixel intensity > 16000 in more than 0.1 fraction of events
status 8: 132 pixel intensity < 1 in more than 0.1 fraction of events
status 16: 0 pixel average > 16000
status 32: 918 pixel average < 11444.5
[I] L0108 status 64: 139 pixel with bad gain mode switch for jungfrau only!

Dark data processing algorithm

Dark data processing algorithm evolved for a long time. The most advanced version, implemented for calibration of jungfrau panels, is described here.

1. dark raw data is accumulated in array block[nrecs1,<2d-shape-of-data>] for a portion of events nrecs1=50. It would be nice to accumulate it for entire volume of events (nrecs=1000), but some of detectors are too big, that causes problem with memory.
2. pre-process data from block, use median and quantile for fraclo=0.05 and fcachi=0.95 fraction of spectral events to estimate mean value and valid gate intensity limits, resperctively.
3. use gated average algorithm to process events in block and all other events (nrecs=1000) requested for dark processing.
4. in summary, evaluate mean, rms, max/min values, and process a few statistical cumulative arrays, which produces information about bad pixels.

Bad pixel status algorithms

For arrays of per-pixel mean intensity and rms, obtained in gated average algorithm, we use parameters of absolute limits and number of sigma

    int_lo=1, int_hi=16000, intnlo=6, intnhi=6

    rms_lo=0.001, rms_hi=16000, rmsnlo=6, rmsnhi=6

and dynamically evaluate limits for good parameters from spectra.

def evaluate_limits(arr, nneg=5, npos=5, lim_lo=1, lim_hi=16000, ...):
    ave, std = (arr.mean(), arr.std())
    lo = ave-nneg*std if nneg>0 else lim_lo
    hi = ave+npos*std if npos>0 else lim_hi
    lo, hi = max(lo, lim_lo), min(hi, lim_hi)
    return lo, hi

Bad pixel bits assignment

status 0: good pixel

status 1:  pixel rms exceeds its maximal value for good pixels defined by rms_hi=16000, rmsnhi=6

status 2: pixel rms lower than its minimal value for good pixels defined by rms_lo=0.001, rmsnlo=6

status 4: pixel intensity exceeds int_hi=16000 in more than fraclm=0.1 fraction of events

status 8: pixel intensity lower than int_lo=1 in more than fraclm=0.1 fraction of events

status 16: pixel average intensity exceeds its maximal value for good pixels defined by int_hi=16000, intnhi=6

status 32: pixel average intensity lower than its minimal value for good pixels defined by int_lo=1, intnlo=6

...

algorithm evolved for a long time. The most advanced version implemented for calibration of jungfrau panels is described here.

1. dark raw data is accumulated in array block[nrecs1,<shape-of-data>] for a portion of events nrecs1=50. It would be nice to accumulate it for entire volume of events (nrecs=1000), but some of detectors are too big and causes problem with memory.
2. pre-process data from block, use median and quantile for fraclo=0.05 and fcachi=0.95 fractions of spectral events to estimate mean value and gate intensity limits, respectively.
3. use gated average algorithm to process events in block and all other events (nrecs=1000) requested for dark processing.
4. at the end of the event loop evaluate mean, rms, max/min values and process a few statistical cumulative arrays which produce information about bad pixels.

Bad pixel status algorithms

For arrays of per-pixel mean intensity and rms, obtained in gated average algorithm, we use parameters of absolute limits and number of sigma for low and hight range

    int_lo=1, int_hi=16000, intnlo=6, intnhi=6

  rms_lo=0.001, rms_hi=16000, rmsnlo=6, rmsnhi=6

and dynamically evaluate limits for good parameters from spectra.

def evaluate_limits(arr, nneg=5, npos=5, lim_lo=1, lim_hi=16000, ...):
    ave, std = (arr.mean(), arr.std())
    lo = ave-nneg*std if nneg>0 else lim_lo
    hi = ave+npos*std if npos>0 else lim_hi
    lo, hi = max(lo, lim_lo), min(hi, lim_hi)
    return lo, hi

Bad pixel bits assignment

status 0: good pixel

status 1:  pixel rms exceeds its maximal value for good pixels defined by rms_hi=16000, rmsnhi=6

status 2: pixel rms lower than its minimal value for good pixels defined by rms_lo=0.001, rmsnlo=6

status 4: pixel intensity exceeds int_hi=16000 in more than fraclm=0.1 fraction of events

status 8: pixel intensity lower than int_lo=1 in more than fraclm=0.1 fraction of events

status 16: pixel average intensity exceeds its maximal value for good pixels defined by int_hi=16000, intnhi=6

status 32: pixel average intensity lower than its minimal value for good pixels defined by int_lo=1, intnlo=6

status 64: for jungfrau only pixel with bad gain mode switch.

Bad pixel statistics

Dark data processing scripts dump in the log file information about bad pixel statistics, for example in case of jungfrau:

raw data found/selected in 999 events
[I] L0688 begin data summary stage
[I] L0111 evaluate_limits RMS: ave=4.783 std=1.303 limits low=0.001 high=12.604
[I] L0111 evaluate_limits AVE: ave=14191.873 std=457.903 limits low=11444.455 high=16000.000

[I] L0733 bad pixel status:
status 1: 244 pixel rms > 12.604
status 2: 550 pixel rms < 0.001
status 4: 418 pixel intensity > 16000 in more than 0.1 fraction of events
status 8: 132 pixel intensity < 1 in more than 0.1 fraction of events
status 16: 0 pixel average > 16000
status 32: 918 pixel average < 11444.5
[I] L0108 status 64: 139 pixel with bad gain mode switch for jungfrau only!

Finding Bad Pixel status information

...