Issue with biased pedestals

Browser images using command

.../psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -Sraw-peds -J1

which plots epixquad images for raw - pedestals

99% and 1% images:

Both type of images can be passed by the gated average algorithm..., but second type gives significant offset to the average value and increases rms.

Finding the list of bad events in the dark processing script (0, 73, 81, 114, 207, 392, 513, 679, 704, 724)

test ouput in the dark processing script 
XXX nrec:000 median(raw-ave): -29.000
XXX nrec:001 median(raw-ave): -5.000
XXX nrec:002 median(raw-ave): -2.000
XXX nrec:003 median(raw-ave): 0.000
XXX nrec:004 median(raw-ave): 0.000

XXX nrec:072 median(raw-ave): 0.000
XXX nrec:073 median(raw-ave): -26.000
XXX nrec:074 median(raw-ave): -6.000
XXX nrec:075 median(raw-ave): -1.000
XXX nrec:076 median(raw-ave): -1.000
XXX nrec:077 median(raw-ave): 0.000
XXX nrec:078 median(raw-ave): 0.000
XXX nrec:079 median(raw-ave): 0.000
XXX nrec:080 median(raw-ave): 0.000
XXX nrec:081 median(raw-ave): -27.000
XXX nrec:082 median(raw-ave): -6.000
XXX nrec:083 median(raw-ave): -1.000
XXX nrec:084 median(raw-ave): 0.000
XXX nrec:085 median(raw-ave): 0.000

XXX nrec:113 median(raw-ave): 0.000
XXX nrec:114 median(raw-ave): -27.000
XXX nrec:115 median(raw-ave): -6.000
XXX nrec:116 median(raw-ave): -2.000
XXX nrec:117 median(raw-ave): -1.000
XXX nrec:118 median(raw-ave): 0.000
XXX nrec:119 median(raw-ave): 0.000

XXX nrec:206 median(raw-ave): 0.000
XXX nrec:207 median(raw-ave): -26.000
XXX nrec:208 median(raw-ave): -5.000
XXX nrec:209 median(raw-ave): -1.000
XXX nrec:210 median(raw-ave): 0.000
XXX nrec:211 median(raw-ave): 0.000

XXX nrec:391 median(raw-ave): 0.000
XXX nrec:392 median(raw-ave): -26.000
XXX nrec:393 median(raw-ave): -6.000
XXX nrec:394 median(raw-ave): -1.000
XXX nrec:395 median(raw-ave): -1.000
XXX nrec:396 median(raw-ave): 0.000
XXX nrec:397 median(raw-ave): 1.000

XXX nrec:512 median(raw-ave): 1.000
XXX nrec:513 median(raw-ave): -26.000
XXX nrec:514 median(raw-ave): -6.000
XXX nrec:515 median(raw-ave): -1.000
XXX nrec:516 median(raw-ave): 0.000
XXX nrec:517 median(raw-ave): 0.000

XXX nrec:678 median(raw-ave): 0.000
XXX nrec:679 median(raw-ave): -26.000
XXX nrec:680 median(raw-ave): -5.000
XXX nrec:681 median(raw-ave): -1.000
XXX nrec:682 median(raw-ave): -1.000
XXX nrec:683 median(raw-ave): 0.000
XXX nrec:684 median(raw-ave): 1.000

XXX nrec:703 median(raw-ave): 0.000
XXX nrec:704 median(raw-ave): -27.000
XXX nrec:705 median(raw-ave): -6.000
XXX nrec:706 median(raw-ave): -1.000
XXX nrec:707 median(raw-ave): 0.000

XXX nrec:723 median(raw-ave): 0.000
XXX nrec:724 median(raw-ave): -26.000
XXX nrec:725 median(raw-ave): -5.000
XXX nrec:726 median(raw-ave): -1.000
XXX nrec:727 median(raw-ave): 0.000
XXX nrec:728 median(raw-ave): 1.000


Results with upgraded algorithm for pedestals calibration

Dark run 101 old vs new pedestals comparison

This is a demo of how user might be easily screwed up applying their own algorithm for pedestal calculation.

Five images of the raw-peds for the same 5-step dark run 101 were obtained by command like 

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M2 -o 2021-03-12-figs/r101-raw-peds-old-dark-ev0510-st2-gn2

then old constants were removed from DB with calibman

new constants were generated with new dark processing algorithm and similar images for new pedestals were generated:

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M2 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st2-gn2

Images of raw-peds in pairs for old and new constants for FH, FM, FL, AHL-H, and AML-M:

FH step 0

FM step 1

FL step2

AHL-H step3

AML-M step 4

Dark run 101 using pedestals from wrong gain range

Another demo of how user can be screwed-up using pedestals from wrong gain range.

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g0 -M2 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st2-gn0

raw(FL) - peds(FH)

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M0 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st1-gn2

raw(FH) - peds(FL)

Dark run 168 with single step in FL

commands:

python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r168-raw-peds-FL-new

python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Scalibcm8 -o 2021-02-05-figs/r168-calibcm8-FL-new

python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Scalib -o 2021-02-05-figs/r168-calib-FL-new

Images:

  • raw-pedestals(FL) old
  • raw-pedestals(FL) new
  • calib
  • calib with cmpars = (8,7,50,10) the same works with cmpars = (8,7,10,10) but not for outlying events where correction is larger than 10 ADU...

Dark 5-step dark run 134

Old pedestals

  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Sraw-peds -M2 -g2 -o 2021-02-05-figs/r134-raw-peds-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalib -M2 -o 2021-02-05-figs/r134-calib-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N1 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL-ev1
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL etc...

Pedestals calibration and deployment commands

  • epix10ka_pedestals_calibration -e ueddaq02 -d epixquad -r134
  • epix10ka_deploy_constants -e ueddaq02 -d epixquad -r134 -D
  • use calibman to remove old constants for run 134

New pedestals

  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Sraw-peds -M2 -g2 -o 2021-02-05-figs/r134-raw-peds-new-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalib -M2 -o 2021-02-05-figs/r134-calib-new-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N1 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL-ev1-new
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-new-FL

Consitency check

Runs 134, 166, 211 cumulative image of 1000 frames of raw-peds descarding events with median intensity 5% in low and high tails:

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -Sraw-peds -g2 -M2 -J1 -C --thrmin -0.344 --thrmax 0.582 --thrpix -10000 -N1000

Mean value of all images is consistent with 0. Visible variation of color causes by variation of intensity spread along surface of the detecor.


1-step FL signal run 137

Old pedestals

  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r137-raw-peds-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalib -o 2021-02-05-figs/r137-calib-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-7-10-10-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-6-10-10-FL (script edited for cmpars...)
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-4-10-10-FL (script edited for cmpars...)

New pedestals

  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r137-raw-peds-new-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalib -o 2021-02-05-figs/r137-calib-new-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-7-10-10-new-FL
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-6-10-10-new-FL (script edited for cmpars...)
  • python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-4-10-10-new-FL (script edited for cmpars...)

Threshold 100 keV

cumulative 1000 events raw-peds with pixel intensit threshold 20,50,100ADU:

Summary for runs 134 and 137

  • pedestals
    • In runs 134 (dark) and 137 (signal) the difference between old and new pedestals calibration algorithm is indistinguishable. This means that run 134 does not have significant bias of pedestals due to outlying events.
    • plots for (raw-peds) and calib are different due to the gain factor only
  • common mode correction
    • plays important role to eliminate intensity fluctuation in banks
    • common mode correction for short rows can easily overcorrect data in bright signal region
  • noise seems different in "left" and "right" ASICs
  • event with threshold 100 keV in run 137 with new pedestals does not show entire banks ..., although presentation might be different

Comparison raw-peds for dark and signal

Dark run 134 and signal run 137 ueddaq02

Intensity limits are selected to discard 1% of events in both side tails.

Dark run 134 step 2-FL normal event 30

  • mean ~0 ADU

Dark run 134 step 2-FL outlier event 0

  • mean ~35 ADU

Signal run 137 (step 0) in FL normal event 30

  • mean ~6 ADU

Signal run 137 (step 0) in FL outlier event 32

  • mean ~-20 ADU

 

Issues for run 137

  • mean value rising during 1000 events of the run 137 step 2/FL
  • pedestals for dark are evaluated correctly
  • baseline for signal has an offset ~5ADU relative to dark
  • there are outliers in signal too which shift mean ~ -25 ADU

Dark run 166 vs signal run 167 ueddaq02

For all events plot raw-pedestals (FL)

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r166 -Sraw-peds -J1 -M2 -K90 -N92 -o 2021-03-08-figs/r166-raw-peds-dark-outlier-ev0092

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r166 -Sraw-peds -J1 -M2 -K90 -N100 -o 2021-03-08-figs/r166-raw-peds-dark-normal-ev0100

Run 166 dark events 100 and 92

run166 event 100 - norman, mean ~ 0 ADU

run 166 event 92 - outlier, mean ~ -25 ADU

Run 167 presumably signal but not so much intensity events 297-300

Commands:

  • lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N297 -o 2021-03-08-figs/r167-raw-peds-signal-normal-ev0297
  • lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N298 -o 2021-03-08-figs/r167-raw-peds-signal-outlier-ev0298
  • lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N299 -o 2021-03-08-figs/r167-raw-peds-signal-outlier-ev0299
  • lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N300 -o 2021-03-08-figs/r167-raw-peds-signal-normal-ev0300

run 167 event 297 - normal, mean ~ 0 ADU

run 167 event 298 - outlier, mean ~ -25 ADU

run 167 event 299 - outlier "recovering", mean ~ -4 ADU

run 167 event 300 - normal, mean ~ 0 ADU

Dark run 263 vs signal run 264 in FH ueddaq02

For all events plot raw-pedestals  in unusual gain mode FH

Dark run 263

events 200-205, cumulative image of 1000 events, and bad event 94

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r263 -Sraw-peds -J1 -K190 -N200 -g0 -M0 -o 2021-03-10-figs/r263-raw-peds-dark-gr0-step0-ev0200

Signal run 264 in gain mode FH

events 0-6, bad event 0

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r264 -Sraw-peds -J1 -N6 --gramin -20 --gramax 80 -o 2021-03-10-figs/r264-raw-peds-signal-ev0006


events 300-304, bad event 302

lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r264 -Sraw-peds -J1 -K299 -N300 --gramin -20 --gramax 80 -o 2021-03-10-figs/r264-raw-peds-signal-ev0300

Issues for runs 263 and 264 FH

  • in 263 mean(raw-peds) <0 for all events ~300
    1. mean intensity raising during the run, see cumulative image of 1000 events
    2. limits on intensity are different comparing to dark processing
  • event-by-event variation of the image mean intensity is larger in FH than in FL
  • outliers are still there both in dark and signal runs
  • in 264 signal is well separated from base level

Correction which solves jitter of the mean value

ueddaq quad shows fluctuation of the baseline-mean image intensity from event to event.

Correction like common mode applied to entire image would solve this problem:

correction for base-line jitter
arr = det.raw.calib(evt)

med = np.median(arr)
if fabs(med>thr_med_max): continue # discard deviation of median exceeding some threshold
arr -= med
<here apply per-pixel thresholds etc to arr>

# generate image from arr, if necessary
img = det.raw.image(evt, nda=arr)










  • No labels