Page History
Table of Contents |
---|
Issue with biased pedestals
Browser images using command
...
which plots epixquad images for raw - pedestals
99% and 1% images:
Both type of images can be passed by the gated average algorithm..., but second type gives significant offset to the average value and increases rms.
Finding the list of bad events in the dark processing script (0, 73, 81, 114, 207, 392, 513, 679, 704, 724)
Code Block | ||||
---|---|---|---|---|
| ||||
XXX nrec:000 median(raw-ave): -29.000
XXX nrec:001 median(raw-ave): -5.000
XXX nrec:002 median(raw-ave): -2.000
XXX nrec:003 median(raw-ave): 0.000
XXX nrec:004 median(raw-ave): 0.000
XXX nrec:072 median(raw-ave): 0.000
XXX nrec:073 median(raw-ave): -26.000
XXX nrec:074 median(raw-ave): -6.000
XXX nrec:075 median(raw-ave): -1.000
XXX nrec:076 median(raw-ave): -1.000
XXX nrec:077 median(raw-ave): 0.000
XXX nrec:078 median(raw-ave): 0.000
XXX nrec:079 median(raw-ave): 0.000
XXX nrec:080 median(raw-ave): 0.000
XXX nrec:081 median(raw-ave): -27.000
XXX nrec:082 median(raw-ave): -6.000
XXX nrec:083 median(raw-ave): -1.000
XXX nrec:084 median(raw-ave): 0.000
XXX nrec:085 median(raw-ave): 0.000
XXX nrec:113 median(raw-ave): 0.000
XXX nrec:114 median(raw-ave): -27.000
XXX nrec:115 median(raw-ave): -6.000
XXX nrec:116 median(raw-ave): -2.000
XXX nrec:117 median(raw-ave): -1.000
XXX nrec:118 median(raw-ave): 0.000
XXX nrec:119 median(raw-ave): 0.000
XXX nrec:206 median(raw-ave): 0.000
XXX nrec:207 median(raw-ave): -26.000
XXX nrec:208 median(raw-ave): -5.000
XXX nrec:209 median(raw-ave): -1.000
XXX nrec:210 median(raw-ave): 0.000
XXX nrec:211 median(raw-ave): 0.000
XXX nrec:391 median(raw-ave): 0.000
XXX nrec:392 median(raw-ave): -26.000
XXX nrec:393 median(raw-ave): -6.000
XXX nrec:394 median(raw-ave): -1.000
XXX nrec:395 median(raw-ave): -1.000
XXX nrec:396 median(raw-ave): 0.000
XXX nrec:397 median(raw-ave): 1.000
XXX nrec:512 median(raw-ave): 1.000
XXX nrec:513 median(raw-ave): -26.000
XXX nrec:514 median(raw-ave): -6.000
XXX nrec:515 median(raw-ave): -1.000
XXX nrec:516 median(raw-ave): 0.000
XXX nrec:517 median(raw-ave): 0.000
XXX nrec:678 median(raw-ave): 0.000
XXX nrec:679 median(raw-ave): -26.000
XXX nrec:680 median(raw-ave): -5.000
XXX nrec:681 median(raw-ave): -1.000
XXX nrec:682 median(raw-ave): -1.000
XXX nrec:683 median(raw-ave): 0.000
XXX nrec:684 median(raw-ave): 1.000
XXX nrec:703 median(raw-ave): 0.000
XXX nrec:704 median(raw-ave): -27.000
XXX nrec:705 median(raw-ave): -6.000
XXX nrec:706 median(raw-ave): -1.000
XXX nrec:707 median(raw-ave): 0.000
XXX nrec:723 median(raw-ave): 0.000
XXX nrec:724 median(raw-ave): -26.000
XXX nrec:725 median(raw-ave): -5.000
XXX nrec:726 median(raw-ave): -1.000
XXX nrec:727 median(raw-ave): 0.000
XXX nrec:728 median(raw-ave): 1.000
|
Results with upgraded algorithm for pedestals calibration
Dark run 101 old vs new pedestals comparison
This is a demo of how user might be easily screwed up applying their own algorithm for pedestal calculation.
Five images of the raw-peds for the same 5-step dark run 101 were obtained by command like
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M2 -o 2021-03-12-figs/r101-raw-peds-old-dark-ev0510-st2-gn2
then old constants were removed from DB with calibman
new constants were generated with new dark processing algorithm and similar images for new pedestals were generated:
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M2 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st2-gn2
Images of raw-peds in pairs for old and new constants for FH, FM, FL, AHL-H, and AML-M:
FH step 0
FM step 1
FL step2
AHL-H step3
AML-M step 4
Dark run 101 using pedestals from wrong gain range
Another demo of how user can be screwed-up using pedestals from wrong gain range.
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g0 -M2 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st2-gn0
raw(FL) - peds(FH)
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r101 -Sraw-peds -J1 -K500 -N510 -g2 -M0 -o 2021-03-12-figs/r101-raw-peds-new-dark-ev0510-st1-gn2
raw(FH) - peds(FL)
Dark run 168 with single step in FL
commands:
python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r168-raw-peds-FL-new
python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Scalibcm8 -o 2021-02-05-figs/r168-calibcm8-FL-new
python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r168 -N100 -Scalib -o 2021-02-05-figs/r168-calib-FL-new
Images:
- raw-pedestals(FL) old
- raw-pedestals(FL) new
- calib
- calib with cmpars = (8,7,50,10) the same works with cmpars = (8,7,10,10) but not for outlying events where correction is larger than 10 ADU...
Dark 5-step dark run 134
Old pedestals
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Sraw-peds -M2 -g2 -o 2021-02-05-figs/r134-raw-peds-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalib -M2 -o 2021-02-05-figs/r134-calib-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N1 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL-ev1
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL etc...
Pedestals calibration and deployment commands
- epix10ka_pedestals_calibration -e ueddaq02 -d epixquad -r134
- epix10ka_deploy_constants -e ueddaq02 -d epixquad -r134 -D
- use calibman to remove old constants for run 134
New pedestals
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Sraw-peds -M2 -g2 -o 2021-02-05-figs/r134-raw-peds-new-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalib -M2 -o 2021-02-05-figs/r134-calib-new-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N1 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-FL-ev1-new
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -N100 -Scalibcm8 -M2 -o 2021-02-05-figs/r134-calibcm8-7-10-10-new-FL
Consitency check
Runs 134, 166, 211 cumulative image of 1000 frames of raw-peds descarding events with median intensity 5% in low and high tails:
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r134 -Sraw-peds -g2 -M2 -J1 -C --thrmin -0.344 --thrmax 0.582 --thrpix -10000 -N1000
Mean value of all images is consistent with 0. Visible variation of color causes by variation of intensity spread along surface of the detecor.
1-step FL signal run 137
Old pedestals
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r137-raw-peds-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalib -o 2021-02-05-figs/r137-calib-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-7-10-10-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-6-10-10-FL (script edited for cmpars...)
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-4-10-10-FL (script edited for cmpars...)
New pedestals
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Sraw-peds -g2 -o 2021-02-05-figs/r137-raw-peds-new-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalib -o 2021-02-05-figs/r137-calib-new-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-7-10-10-new-FL
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-6-10-10-new-FL (script edited for cmpars...)
- python lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r137 -N100 -Scalibcm8 -o 2021-02-05-figs/r137-calibcm8-4-10-10-new-FL (script edited for cmpars...)
Threshold 100 keV
cumulative 1000 events raw-peds with pixel intensit threshold 20,50,100ADU:
Summary for runs 134 and 137
- pedestals
- In runs 134 (dark) and 137 (signal) the difference between old and new pedestals calibration algorithm is indistinguishable. This means that run 134 does not have significant bias of pedestals due to outlying events.
- plots for (raw-peds) and calib are different due to the gain factor only
- common mode correction
- plays important role to eliminate intensity fluctuation in banks
- common mode correction for short rows can easily overcorrect data in bright signal region
- noise seems different in "left" and "right" ASICs
- event with threshold 100 keV in run 137 with new pedestals does not show entire banks ..., although presentation might be different
Comparison raw-peds for dark and signal
Dark run 134 and signal run 137 ueddaq02
Intensity limits are selected to discard 1% of events in both side tails.
Dark run 134 step 2-FL normal event 30
- mean ~0 ADU
Dark run 134 step 2-FL outlier event 0
- mean ~35 ADU
Signal run 137 (step 0) in FL normal event 30
- mean ~6 ADU
Signal run 137 (step 0) in FL outlier event 32
- mean ~-20 ADU
Issues for run 137
- mean value rising during 1000 events of the run 137 step 2/FL
- pedestals for dark are evaluated correctly
- baseline for signal has an offset ~5ADU relative to dark
- there are outliers in signal too which shift mean ~ -25 ADU
Dark run 166 vs signal run 167 ueddaq02
For all events plot raw-pedestals (FL)
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r166 -Sraw-peds -J1 -M2 -K90 -N92 -o 2021-03-08-figs/r166-raw-peds-dark-outlier-ev0092
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r166 -Sraw-peds -J1 -M2 -K90 -N100 -o 2021-03-08-figs/r166-raw-peds-dark-normal-ev0100
Run 166 dark events 100 and 92
run166 event 100 - norman, mean ~ 0 ADU
run 166 event 92 - outlier, mean ~ -25 ADU
Run 167 presumably signal but not so much intensity events 297-300
Commands:
- lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N297 -o 2021-03-08-figs/r167-raw-peds-signal-normal-ev0297
- lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N298 -o 2021-03-08-figs/r167-raw-peds-signal-outlier-ev0298
- lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N299 -o 2021-03-08-figs/r167-raw-peds-signal-outlier-ev0299
- lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r167 -Sraw-peds -J1 -K295 -N300 -o 2021-03-08-figs/r167-raw-peds-signal-normal-ev0300
run 167 event 297 - normal, mean ~ 0 ADU
run 167 event 298 - outlier, mean ~ -25 ADU
run 167 event 299 - outlier "recovering", mean ~ -4 ADU
run 167 event 300 - normal, mean ~ 0 ADU
Dark run 263 vs signal run 264 in FH ueddaq02
For all events plot raw-pedestals in unusual gain mode FH
Dark run 263
events 200-205, cumulative image of 1000 events, and bad event 94
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r263 -Sraw-peds -J1 -K190 -N200 -g0 -M0 -o 2021-03-10-figs/r263-raw-peds-dark-gr0-step0-ev0200
Signal run 264 in gain mode FH
events 0-6, bad event 0
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r264 -Sraw-peds -J1 -N6 --gramin -20 --gramax 80 -o 2021-03-10-figs/r264-raw-peds-signal-ev0006
events 300-304, bad event 302
lcls2/psana/psana/detector/test_01_epix10ka_raw_calib_image.py image -e ueddaq02 -d epixquad -r264 -Sraw-peds -J1 -K299 -N300 --gramin -20 --gramax 80 -o 2021-03-10-figs/r264-raw-peds-signal-ev0300
Issues for runs 263 and 264 FH
- in 263 mean(raw-peds) <0 for all events ~300
- mean intensity raising during the run, see cumulative image of 1000 events
- limits on intensity are different comparing to dark processing
- event-by-event variation of the image mean intensity is larger in FH than in FL
- outliers are still there both in dark and signal runs
- in 264 signal is well separated from base level
Correction which solves jitter of the mean value
ueddaq quad shows fluctuation of the baseline-mean image intensity from event to event.
Correction like common mode applied to entire image would solve this problem:
Code Block | ||
---|---|---|
| ||
arr = det.raw.calib(evt)
med = np.median(arr)
if fabs(med>thr_med_max): continue # discard deviation of median exceeding some threshold
arr -= med
<here apply per-pixel thresholds etc to arr>
# generate image from arr, if necessary
img = det.raw.image(evt, nda=arr) |