- Created by Mikhail Dubrovin, last modified on May 12, 2023
Issue
e-mail from Chuck
Yoon, Chun Hong <yoon82@slac.stanford.edu> 2023-01-24 10:13pm Hi Mikhail, I came across a user working on xpplw3319 dataset. It seemed that our LCLS mask was not sufficient for their analysis. Please see attached 4 detector averaged over multiple events and 4 detector masks (run293, epic_alc1,2,3,4). Do you think setting more conservative parameters in selecting bad pixel map would help in this case? Best, Chuck
Dataset
xpplw3319 run 293, dark run 287
datinfo -e xpplw3319 -r 293 -d epix_alc1
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.1)', alias='epix_alc1')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.2)', alias='epix_alc2')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.3)', alias='epix_alc3')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.4)', alias='epix_alc4')
[I] L0120: calibDir: /cds/data/psdm/xpp/xpplw3319/calib
Average image and mask for four detectors
Dark processing check
https://pswww.slac.stanford.edu/lgbk/lgbk/xpplw3319/#eLog
Jul/16/2021 06:11:15 287 dark - Restored
Jul/16/2021 07:44:51 293
===
Restore 287 dark
/cds/data/psdm/xpp/xpplw3319/xtc/xpplw3319-r028*
Command and logfiles for four detectors
det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1
logfile for det1 Expand source
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000
[I] UtilsCalib.py L0327 record at start:
2023-01-26T09:13:07 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000
saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt
[I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000
Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.1', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=287 None
source XppGon.0:Epix100a.1 None
ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt
events 1000 1000
evskip 0 0
intlow 1 1
inthig 16000 16000
rmslow 0.001 0.001
rmshig 16000 16000
fraclm 0.1 0.1
nsigma 6.0 6.0
plotim 0 0
verbos 7 7
savebw 255 255
intnlo 6.0 6.0
intnhi 6.0 6.0
rmsnlo 6.0 6.0
rmsnhi 6.0 6.0
evcode None None
dirrepo work ./work
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
nrecs 1000 100
nrecs1 50 50
[I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287
[I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.1
[I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...]
[I] UtilsDarkProc.py L0168 Event: 0, time= 0.127 sec, dt=0.127 sec
[I] UtilsDarkProc.py L0168 Event: 1, time= 0.130 sec, dt=0.003 sec
[I] UtilsDarkProc.py L0168 Event: 2, time= 0.135 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 3, time= 0.139 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 4, time= 0.144 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 10, time= 0.171 sec, dt=0.027 sec
[I] UtilsDarkProc.py L0168 Event: 20, time= 0.193 sec, dt=0.023 sec
[I] UtilsDarkProc.py L0168 Event: 30, time= 0.214 sec, dt=0.021 sec
[I] UtilsDarkProc.py L0168 Event: 40, time= 0.233 sec, dt=0.020 sec
[I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [3761 3889 3838 3956 3827...]
[I] UtilsCalib.py L0587 proc_block pre-processing time 2.491 sec
results for median over pixels intensities:
0.500 fraction of the event spectrum is below 3920.000 ADU - pedestal estimator
0.050 fraction of the event spectrum is below 3916.000 ADU - gate low limit
0.950 fraction of the event spectrum is below 3925.000 ADU - gate upper limit
event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator
[I] UtilsCalib.py L0641 data block processing total time 2.514 sec
arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [3758. 3885. 3839. 3955. 3828....]
abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [1.5 1. 2. 2. 2. ...]
gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [3754 3882 3836 3951 3823...]
gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [3762 3889 3845 3960 3833...]
[I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16
[I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4045 4231 4160 4173 3957...]
1st stage event block processing is completed
[I] UtilsDarkProc.py L0168 Event: 100, time= 4.665 sec, dt=4.432 sec
[I] UtilsDarkProc.py L0168 Event: 200, time= 6.616 sec, dt=1.952 sec
[I] UtilsDarkProc.py L0168 Event: 300, time= 8.555 sec, dt=1.939 sec
[I] UtilsDarkProc.py L0168 Event: 400, time= 10.518 sec, dt=1.962 sec
[I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000
[I] UtilsCalib.py L0679 summary
[I] UtilsCalib.py L0680 ________________________________________________________________________________
raw data found/selected in 1000 events
[I] UtilsCalib.py L0683 begin data summary stage
[I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.203 std=0.445 limits low=0.001 high=4.872
[I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3900.681 std=209.658 limits low=2642.732 high=5158.629
[I] UtilsCalib.py L0721 bad pixel status:
status 1: 379 pixel rms > 4.872
status 2: 0 pixel rms < 0.001
status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events
status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events
status 16: 15 pixel average > 5158.63
status 32: 0 pixel average < 2642.73
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
[I] UtilsDarkProc.py L0174 ________________________________________________________________________________
Total consumed time = 22.977433 sec.
logfile for det2 Expand source
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000
[I] UtilsCalib.py L0327 record at start:
2023-01-26T09:17:49 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000
saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt
[I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000
Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.2', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=287 None
source XppGon.0:Epix100a.2 None
ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt
events 1000 1000
evskip 0 0
intlow 1 1
inthig 16000 16000
rmslow 0.001 0.001
rmshig 16000 16000
fraclm 0.1 0.1
nsigma 6.0 6.0
plotim 0 0
verbos 7 7
savebw 255 255
intnlo 6.0 6.0
intnhi 6.0 6.0
rmsnlo 6.0 6.0
rmsnhi 6.0 6.0
evcode None None
dirrepo work ./work
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
nrecs 1000 100
nrecs1 50 50
[I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287
[I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.2
[I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...]
[I] UtilsDarkProc.py L0168 Event: 0, time= 0.085 sec, dt=0.085 sec
[I] UtilsDarkProc.py L0168 Event: 1, time= 0.089 sec, dt=0.003 sec
[I] UtilsDarkProc.py L0168 Event: 2, time= 0.093 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 3, time= 0.098 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 4, time= 0.102 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 10, time= 0.128 sec, dt=0.026 sec
[I] UtilsDarkProc.py L0168 Event: 20, time= 0.151 sec, dt=0.023 sec
[I] UtilsDarkProc.py L0168 Event: 30, time= 0.172 sec, dt=0.021 sec
[I] UtilsDarkProc.py L0168 Event: 40, time= 0.192 sec, dt=0.021 sec
[I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [4154 4215 4185 4397 4179...]
[I] UtilsCalib.py L0587 proc_block pre-processing time 2.500 sec
results for median over pixels intensities:
0.500 fraction of the event spectrum is below 4072.000 ADU - pedestal estimator
0.050 fraction of the event spectrum is below 4068.000 ADU - gate low limit
0.950 fraction of the event spectrum is below 4076.000 ADU - gate upper limit
event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator
[I] UtilsCalib.py L0641 data block processing total time 2.524 sec
arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [4156. 4217. 4181. 4397. 4181....]
abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [2. 2. 2. 2. 2....]
gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [4151 4213 4178 4393 4175...]
gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [4159 4222 4186 4400 4184...]
[I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16
[I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4300 4651 4343 4538 4177...]
1st stage event block processing is completed
[I] UtilsDarkProc.py L0168 Event: 100, time= 4.653 sec, dt=4.461 sec
[I] UtilsDarkProc.py L0168 Event: 200, time= 6.686 sec, dt=2.033 sec
[I] UtilsDarkProc.py L0168 Event: 300, time= 8.654 sec, dt=1.969 sec
[I] UtilsDarkProc.py L0168 Event: 400, time= 10.682 sec, dt=2.028 sec
[I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000
[I] UtilsCalib.py L0679 summary
[I] UtilsCalib.py L0680 ________________________________________________________________________________
raw data found/selected in 1000 events
[I] UtilsCalib.py L0683 begin data summary stage
[I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.330 std=0.995 limits low=0.001 high=8.303
[I] UtilsCalib.py L0112 evaluate_limits AVE: ave=4065.543 std=226.214 limits low=2708.260 high=5422.827
[I] UtilsCalib.py L0721 bad pixel status:
status 1: 239 pixel rms > 8.303
status 2: 0 pixel rms < 0.001
status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events
status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events
status 16: 26 pixel average > 5422.83
status 32: 1 pixel average < 2708.26
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
[I] UtilsDarkProc.py L0174 ________________________________________________________________________________
Total consumed time = 23.291649 sec.
[I] det_ndarr_raw_proc L0141 log file: None
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
logfile for det3 Expand source
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000
[I] UtilsCalib.py L0327 record at start:
2023-01-26T09:21:06 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000
saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt
[I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000
Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.3', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=287 None
source XppGon.0:Epix100a.3 None
ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt
events 1000 1000
evskip 0 0
intlow 1 1
inthig 16000 16000
rmslow 0.001 0.001
rmshig 16000 16000
fraclm 0.1 0.1
nsigma 6.0 6.0
plotim 0 0
verbos 7 7
savebw 255 255
intnlo 6.0 6.0
intnhi 6.0 6.0
rmsnlo 6.0 6.0
rmsnhi 6.0 6.0
evcode None None
dirrepo work ./work
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
nrecs 1000 100
nrecs1 50 50
[I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287
[I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.3
[I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...]
[I] UtilsDarkProc.py L0168 Event: 0, time= 0.074 sec, dt=0.074 sec
[I] UtilsDarkProc.py L0168 Event: 1, time= 0.078 sec, dt=0.003 sec
[I] UtilsDarkProc.py L0168 Event: 2, time= 0.082 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 3, time= 0.087 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 4, time= 0.091 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 10, time= 0.118 sec, dt=0.027 sec
[I] UtilsDarkProc.py L0168 Event: 20, time= 0.141 sec, dt=0.023 sec
[I] UtilsDarkProc.py L0168 Event: 30, time= 0.161 sec, dt=0.021 sec
[I] UtilsDarkProc.py L0168 Event: 40, time= 0.183 sec, dt=0.022 sec
[I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [3841 3962 3836 3825 3845...]
[I] UtilsCalib.py L0587 proc_block pre-processing time 2.515 sec
results for median over pixels intensities:
0.500 fraction of the event spectrum is below 3982.000 ADU - pedestal estimator
0.050 fraction of the event spectrum is below 3977.000 ADU - gate low limit
0.950 fraction of the event spectrum is below 3987.000 ADU - gate upper limit
event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator
[I] UtilsCalib.py L0641 data block processing total time 2.537 sec
arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [3844. 3963. 3839. 3831.5 3845. ...]
abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [2. 2. 2. 1.5 2. ...]
gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [3837 3958 3835 3825 3840...]
gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [3847 3969 3843 3836 3850...]
[I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16
[I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [3632 3974 3821 3918 3696...]
1st stage event block processing is completed
[I] UtilsDarkProc.py L0168 Event: 100, time= 4.701 sec, dt=4.518 sec
[I] UtilsDarkProc.py L0168 Event: 200, time= 6.827 sec, dt=2.126 sec
[I] UtilsDarkProc.py L0168 Event: 300, time= 8.874 sec, dt=2.047 sec
[I] UtilsDarkProc.py L0168 Event: 400, time= 10.912 sec, dt=2.038 sec
[I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000
[I] UtilsCalib.py L0679 summary
[I] UtilsCalib.py L0680 ________________________________________________________________________________
raw data found/selected in 1000 events
[I] UtilsCalib.py L0683 begin data summary stage
[I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.317 std=0.370 limits low=0.097 high=4.536
[I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3983.565 std=188.830 limits low=2850.584 high=5116.546
[I] UtilsCalib.py L0721 bad pixel status:
status 1: 643 pixel rms > 4.536
status 2: 0 pixel rms < 0.097
status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events
status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events
status 16: 31 pixel average > 5116.55
status 32: 0 pixel average < 2850.58
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
[I] UtilsDarkProc.py L0174 ________________________________________________________________________________
Total consumed time = 23.812071 sec.
[I] det_ndarr_raw_proc L0141 log file: None
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
logfile for det4 Expand source
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000
[I] UtilsCalib.py L0327 record at start:
2023-01-26T09:22:44 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000
saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt
[I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000
Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.4', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=287 None
source XppGon.0:Epix100a.4 None
ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt
events 1000 1000
evskip 0 0
intlow 1 1
inthig 16000 16000
rmslow 0.001 0.001
rmshig 16000 16000
fraclm 0.1 0.1
nsigma 6.0 6.0
plotim 0 0
verbos 7 7
savebw 255 255
intnlo 6.0 6.0
intnhi 6.0 6.0
rmsnlo 6.0 6.0
rmsnhi 6.0 6.0
evcode None None
dirrepo work ./work
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
nrecs 1000 100
nrecs1 50 50
[I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287
[I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.4
[I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...]
[I] UtilsDarkProc.py L0168 Event: 0, time= 0.074 sec, dt=0.074 sec
[I] UtilsDarkProc.py L0168 Event: 1, time= 0.078 sec, dt=0.003 sec
[I] UtilsDarkProc.py L0168 Event: 2, time= 0.082 sec, dt=0.004 sec
[I] UtilsDarkProc.py L0168 Event: 3, time= 0.086 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 4, time= 0.091 sec, dt=0.005 sec
[I] UtilsDarkProc.py L0168 Event: 10, time= 0.118 sec, dt=0.027 sec
[I] UtilsDarkProc.py L0168 Event: 20, time= 0.141 sec, dt=0.023 sec
[I] UtilsDarkProc.py L0168 Event: 30, time= 0.161 sec, dt=0.021 sec
[I] UtilsDarkProc.py L0168 Event: 40, time= 0.181 sec, dt=0.020 sec
[I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [4523 4558 4216 4521 4508...]
[I] UtilsCalib.py L0587 proc_block pre-processing time 2.555 sec
results for median over pixels intensities:
0.500 fraction of the event spectrum is below 4577.000 ADU - pedestal estimator
0.050 fraction of the event spectrum is below 4571.000 ADU - gate low limit
0.950 fraction of the event spectrum is below 4583.000 ADU - gate upper limit
event spectrum spread median(abs(raw-med)): 2.500 ADU - spectral peak width estimator
[I] UtilsCalib.py L0641 data block processing total time 2.577 sec
arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [4521. 4554. 4220. 4524.5 4508.5...]
abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [1.5 2. 2. 2.5 2.5...]
gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [4516 4550 4215 4519 4505...]
gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [4525 4563 4224 4529 4513...]
[I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16
[I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4626 4797 4659 4839 4565...]
1st stage event block processing is completed
[I] UtilsDarkProc.py L0168 Event: 100, time= 4.722 sec, dt=4.541 sec
[I] UtilsDarkProc.py L0168 Event: 200, time= 6.681 sec, dt=1.959 sec
[I] UtilsDarkProc.py L0168 Event: 300, time= 8.618 sec, dt=1.937 sec
[I] UtilsDarkProc.py L0168 Event: 400, time= 10.753 sec, dt=2.135 sec
[I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000
[I] UtilsCalib.py L0679 summary
[I] UtilsCalib.py L0680 ________________________________________________________________________________
raw data found/selected in 1000 events
[I] UtilsCalib.py L0683 begin data summary stage
[I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.965 std=0.396 limits low=0.587 high=5.344
[I] UtilsCalib.py L0112 evaluate_limits AVE: ave=4581.344 std=278.259 limits low=2911.793 high=6250.895
[I] UtilsCalib.py L0721 bad pixel status:
status 1: 286 pixel rms > 5.344
status 2: 1 pixel rms < 0.587
status 4: 1 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events
status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events
status 16: 373 pixel average > 6250.9
status 32: 1 pixel average < 2911.79
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved
[I] UtilsDarkProc.py L0174 ________________________________________________________________________________
Total consumed time = 23.475768 sec.
[I] det_ndarr_raw_proc L0141 log file: None
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
Dark average (pedestals) and pixel_rms images with spectra
epix_alc1
epix_alc2
epix_alc3
epix_alc4
Interim Summary
- In alc1 two data banks look different from other banks, but they are not so bad to discard them by changing thresholds.
- The "scratch" in alc3 is not seen in dark.
- The number of bad/discarded pixels can be changed by variation of thresholds.
Summary of the mtg with Chuck
Discuss article Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography
- Q: definition of robust parameters
A: robust mean is median xi
robust sigma is median |xi-mu|
- Q: difference in algorithms
A:
- for dark intensity and rms algorithms are about the same
- Philip evaluated gains for a few detectors
- but in general we do not have/use bad pixel status for gain factors (generated from flat field)
WE NEED IN LIGHT FLAT FIELD DATA IN ALL GAIN RANGES ALONG WITH DARK
- for gain and bad pixel status
- Q: What can we do to improve bad pixel mask?
A:
- their code is terrible
- re-implement and try their algorithms
- Q: need help in analysis with the bad pixel mask to estimate its performance
- hit rate
- indexing efficiency
A: - Chuck's team can do that
- Chuck: start with "flat field" sample for epix_alc3 and try to find bad pixels "next to central row and "scratch"
2023-03-09 Implementation of Feature 1 and 6 for data
Data sample and detector for test
- dsname exp=xpplw3319:run=293
- XppGon.0:Epix100a.3
Script and accumulation of data block
Logfile in the beginning of the script Expand source
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
[I] det_raw_pixel_status L0146 log file: work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt
[I] UtilsCalib.py L0327 record at start:
2023-03-09T12:11:09 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt command:Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
saved in: work/2023/2023_lcls1_det_raw_pixel_status.txt
[I] det_raw_pixel_status L0150 Command: Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
Argument list: Namespace(dsname='exp=xpplw3319:run=293', source='epix_alc3', events=1000, evskip=0, steps=None, stskip=None, plotim=0, evcode=None, segind=None, dirrepo='work', nrecs=1000, logmode='INFO', dirmode=1533, filemode=436, group='ps-users', upload=False, slice='0:,0:', shwind='15,20', snrmax=8.0, databits=16383, gainbits=16383)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=293 exp=xpplw3319:run=293
source epix_alc3 epix_alc3
events 1000 100
evskip 0 0
steps None None
stskip None None
plotim 0 0
evcode None None
segind None None
dirrepo work work
nrecs 1000 1000
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
upload False False
slice 0:,0: 0:,0:
shwind 15,20 15,15
snrmax 8.0 8.0
databits 16383 16383
gainbits 0 0
[I] UtilsRawPixelStatus.py L0351 detector type: epix100a id: 3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsRawPixelStatus.py L0361 ==== run 293
[I] UtilsRawPixelStatus.py L0367 == calibcycle 01 ==
[I] UtilsRawPixelStatus.py L0393 Ev:0001 rec:-001 raw shape:(704, 768) size:540672 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0181 created empty data block shape:(1000, 704, 768) size:540672000 dtype:uint16 [0 0 0 0 0...]
[I] UtilsRawPixelStatus.py L0393 Ev:0002 rec:0000 raw shape:(704, 768) size:540672 dtype:uint16 [3622 3968 3818 3917 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0003 rec:0001 raw shape:(704, 768) size:540672 dtype:uint16 [3629 3972 3815 3914 3689...]
[I] UtilsRawPixelStatus.py L0393 Ev:0004 rec:0002 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3970 3819 3918 3688...]
[I] UtilsRawPixelStatus.py L0393 Ev:0005 rec:0003 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3967 3821 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0011 rec:0009 raw shape:(704, 768) size:540672 dtype:uint16 [3625 3970 3819 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0021 rec:0019 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3819 3916 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0031 rec:0029 raw shape:(704, 768) size:540672 dtype:uint16 [3621 3969 3820 3919 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0041 rec:0039 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3817 3920 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0101 rec:0099 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3969 3818 3914 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0201 rec:0199 raw shape:(704, 768) size:540672 dtype:uint16 [3630 3970 3814 3918 3691...]
[I] UtilsRawPixelStatus.py L0393 Ev:0301 rec:0299 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3973 3813 3910 3693...]
[I] UtilsRawPixelStatus.py L0393 Ev:0401 rec:0399 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3977 3821 3918 3695...]
[I] UtilsRawPixelStatus.py L0393 Ev:1001 rec:0999 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3974 3822 3922 3694...]
[I] UtilsRawPixelStatus.py L0305 nevt:1001 accumulated requested number of records --nrecs: 1000 - break
[I] UtilsRawPixelStatus.py L0404 BREAK EVENTS
[I] UtilsRawPixelStatus.py L0414 BREAK STEPS
[I] UtilsRawPixelStatus.py L0418 BREAK RUNS
[I] UtilsRawPixelStatus.py L0246
...
Issue for Feature 1
Logfile for Feat.1 Expand source
Summary for data block: shape:(1000, 704, 768) size:540672000 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0195
block record intensity MEDIAN IN FRAME: shape:(1000,) size:1000 dtype:float64
[3982. 3983. 3981. 3982. 3982. 3981. 3981. 3982. 3983. 3983. 3982. 3981.
3982. 3983. 3982. 3982. 3982. 3981. 3983. 3981....]
[W] UtilsRawPixelStatus.py L0099 MEDIAN OF SPREAD FOR INT VALUES IS 0 replaced with STD/4 = 0.132
[I] UtilsRawPixelStatus.py L0114 ====================
intensity_est shape:(1000,) size:1000 dtype:float64 [...]
0 / 1000 ( 0.000%) pixels intensity_est <= 0.000
0 / 1000 ( 0.000%) pixels intensity_est >= 16383.000
intensity_est selected 1000 of 1000 pixels in range (0, 16383) med: 3982.000 spr: 0.132
re-defined range for med ± 8.0*spr: (3980.946, 3983.054)
0 / 1000 ( 0.000%) pixels intensity_est <= 3980.946
2 / 1000 ( 0.200%) pixels intensity_est >= 3983.054
[I] UtilsRawPixelStatus.py L0199 Total number of good events: 998
[I] UtilsRawPixelStatus.py L0253 arr1_good_frames: shape:(1000,) size:1000 dtype:uint64 [1 1 1 1 1...]
[I] UtilsRawPixelStatus.py L0255 inds_good_frames: shape:(998,) size:998 dtype:int64 [...]
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 98 99 100 101 102 103 104 105 106 107 108
...
- median of frame intensity is int. Variation between frames is very small ~ +/- 1 ADU
- median spread is also int 0
- can't set med +/- 8 spr limits... does not work if implemented as described in Article
-> if spread is 0 it is evaluated as std
Feature 2 and 3 for dark run
Implemented on 2023-04-06 exactly as described. Median is used for robust statistics parameters evaluation.
Feature 4 and 5 for gain modes
TBD
Feature 6
Test of the fit to plane
Logfile for Feat.6 processing Expand source
...
[I] UtilsRawPixelStatus.py L0240 frame: 0991 res_med: 1.282 res_spr: 110.094 frame residuals shape:(704, 768) size:540672 dtype:float64 [-236.87714286 103.30887218 -49.50511278...]
[I] UtilsRawPixelStatus.py L0240 frame: 0992 res_med: 1.294 res_spr: 110.065 frame residuals shape:(704, 768) size:540672 dtype:float64 [-235.11755952 106.95056078 -54.98131892...]
[I] UtilsRawPixelStatus.py L0240 frame: 0993 res_med: 1.253 res_spr: 110.069 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.62863095 108.57688283 -45.21760338...]
[I] UtilsRawPixelStatus.py L0240 frame: 0994 res_med: 1.267 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-238.30857143 107.88390977 -47.92360902...]
[I] UtilsRawPixelStatus.py L0240 frame: 0995 res_med: 1.218 res_spr: 110.053 frame residuals shape:(704, 768) size:540672 dtype:float64 [-239.62184524 108.59725251 -37.18364975...]
[I] UtilsRawPixelStatus.py L0240 frame: 0996 res_med: 1.251 res_spr: 110.064 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.58547619 109.59983709 -43.21484962...]
[I] UtilsRawPixelStatus.py L0240 frame: 0997 res_med: 1.307 res_spr: 110.067 frame residuals shape:(704, 768) size:540672 dtype:float64 [-240.9114881 104.21362469 -50.66126253...]
[I] UtilsRawPixelStatus.py L0240 frame: 0998 res_med: 1.253 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-233.00535714 106.10161028 -47.79142231...]
[I] UtilsRawPixelStatus.py L0240 frame: 0999 res_med: 1.289 res_spr: 110.099 frame residuals shape:(704, 768) size:540672 dtype:float64 [-241.53529762 109.61091792 -48.24286654...]
[I] UtilsRawPixelStatus.py L0265 block of residuals: shape:(998, 704, 768) size:539590656 dtype:float64
[-239.3222619 106.86876566 -48.94020677 48.2508208 -174.55815163
111.63287594 -128.17609649 -21.98506892 -17.79404135 80.39698622
-86.41198622 61.77904135 -116.02993108 118.16109649 -199.64787594
46.54315163 -238.2658208 95.92520677 -148.88376566 16.3072619 ...]
[I] UtilsRawPixelStatus.py L0266 median over frames per-pixel residuals: shape:(704, 768) size:540672 dtype:float64
[-237.54059524 107.78383145 -46.79742951 51.89206767 -174.04665414
109.26853227 -124.40214912 -22.8108537 -14.74468515 80.47136278
-87.92196429 64.17849154 -117.18580984 119.22582237 -198.74469768
49.81222744 -235.53807174 93.36754543 -146.50276942 16.18511905...]
[I] UtilsRawPixelStatus.py L0267 median over frames per-pixel spread of res: shape:(704, 768) size:540672 dtype:float64
[1.91488095 1.90238878 1.80436404 1.79662124 1.58172462 1.79079887
1.86366228 1.83122337 1.75326128 1.94361685 1.74409305 1.90914474
1.7539317 1.96598684 1.92817356 2.05658521 1.84223058 2.07546366
1.7810354 1.90964286...]
[I] UtilsRawPixelStatus.py L0114 ====================
Feat.6 res_med shape:(704, 768) size:540672 dtype:float64 [...]
0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -16383.000
0 / 540672 ( 0.000%) pixels Feat.6 res_med >= 16383.000
Feat.6 res_med selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.225 spr: 110.081
re-defined range for med ± 8.0*spr: (-879.424, 881.874)
0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
[I] UtilsRawPixelStatus.py L0114 ====================
Feat.6 res_spr shape:(704, 768) size:540672 dtype:float64 [...]
0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= -16383.000
0 / 540672 ( 0.000%) pixels Feat.6 res_spr >= 16383.000
Feat.6 res_spr selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.788 spr: 0.105
re-defined range for med ± 8.0*spr: (0.950, 2.626)
0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
[I] UtilsRawPixelStatus.py L0284
Summary of the bad pixel status evaluation, pixel_status_extra array
0o1: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
0o2: 33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
0o4: 0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
0o10: 4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
Any bad status bit: 4882 / 540672 ( 0.903%) pixels
[I] UtilsCalib.py L0150 saved: pixel_status_extra.dat
[I] UtilsRawPixelStatus.py L0428 Consumed time 1831.462 sec
Pixel status after Feature 6 and our regular mask after dark processing and averaged data
- Feat. 6 adds more bad pixels, especially stripe in the lower right ASIC
- "scratch" seen in data is not reproduced
Chuck wants to find "scratch" automatically
2023-04-06 Feature 2 and 3 for dark data vs our standard dark processing
- dark data set: exp=xpplw3319:run=287
- detector: XppGon.0:Epix100a.3, alias: epix_alc3
- This data set contains ~1200 events, of which initial 1000 is used in processing.
- By default features 1,2,3 are used. Feature 4 and 5 for gain factors are not implemented and are irrelevant to epix100a.
- Feature 1 - good frame selection discards 1/0/0 frames for SNR=3/5/8, respectively
Results for SNR max = 3,5,8 Expand source
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R3 --gmode SNR3
Summary of the bad pixel status evaluation for SNR = 3.00, status_data array
0o1: 6445 / 540672 ( 1.192%) pixels Feat.2 mean <= 3588.608
0o2: 13484 / 540672 ( 2.494%) pixels Feat.2 mean >= 4374.797
0o4: 5600 / 540672 ( 1.036%) pixels Feat.3 std <= 2.314
0o10: 34098 / 540672 ( 6.307%) pixels Feat.3 std >= 3.263
Any bad status bit: 57409 / 540672 (10.618%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R5 --gmode SNR5
Summary of the bad pixel status evaluation for SNR=5.00, status_data array
0o1: 5 / 540672 ( 0.001%) pixels Feat.2 mean <= 3326.536
0o2: 87 / 540672 ( 0.016%) pixels Feat.2 mean >= 4636.866
0o4: 878 / 540672 ( 0.162%) pixels Feat.3 std <= 1.999
0o10: 12739 / 540672 ( 2.356%) pixels Feat.3 std >= 3.579
Any bad status bit: 13675 / 540672 ( 2.529%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R8 --gmode SNR8
Summary of the bad pixel status evaluation for SNR=8.00, status_data array
0o1: 0 / 540672 ( 0.000%) pixels Feat.2 mean <= 2933.437
0o2: 33 / 540672 ( 0.006%) pixels Feat.2 mean >= 5029.965
0o4: 0 / 540672 ( 0.000%) pixels Feat.3 std <= 1.525
0o10: 5314 / 540672 ( 0.983%) pixels Feat.3 std >= 4.053
Any bad status bit: 5319 / 540672 ( 0.984%) pixels
pixel_status from regular dark processing Expand source
det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -o work raw data found/selected in 1000 events [I] UtilsCalib.py L0683 begin data summary stage [I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.317 std=0.370 limits low=0.097 high=4.536 [I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3983.565 std=188.830 limits low=2850.584 high=5116.546 [I] UtilsCalib.py L0721 bad pixel status: status 1: 643 pixel rms > 4.536 status 2: 0 pixel rms < 0.097 status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events status 16: 31 pixel average > 5116.55 status 32: 0 pixel average < 2850.58
2023-04-06 Feature 6 for regular data for SNR=3 and 5
- dark data set: exp=xpplw3319:run=287
- detector: XppGon.0:Epix100a.3, alias: epix_alc3
Feature 6 for SNR=3 and 5 Expand source
det_raw_pixel_status -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 1000 -R3 --gmode SNR3-F6 -F 1,6
Summary of the bad pixel status evaluation for SNR=3.00, status_data array
0o1: 449 / 540672 ( 0.083%) pixels Feat.6 res_med <= -330.137
0o2: 379 / 540672 ( 0.070%) pixels Feat.6 res_med >= 331.245
0o4: 8305 / 540672 ( 1.536%) pixels Feat.6 res_spr <= 1.472
0o10: 34625 / 540672 ( 6.404%) pixels Feat.6 res_spr >= 2.095
Any bad status bit: 43648 / 540672 ( 8.073%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 1000 -R5 --gmode SNR5-F6 -F 1,6
Summary of the bad pixel status evaluation for SNR=5.00, status_data array
0o1: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -550.599
0o2: 45 / 540672 ( 0.008%) pixels Feat.6 res_med >= 551.708
0o4: 1005 / 540672 ( 0.186%) pixels Feat.6 res_spr <= 1.265
0o10: 12484 / 540672 ( 2.309%) pixels Feat.6 res_spr >= 2.302
Any bad status bit: 13501 / 540672 ( 2.497%) pixels
Interim summary
- Number of bad pixels depends on SNR parameter
- Stripe of bad pixels in the middle of the detector can be caught by Feature 3 and 6 at SNR=5.
- "Scratch" observed by Chuck is not seen detected ...
2023-04-07 mtg summary
- Chuck: try to search for scratch in ASIC ONLY!
- Chris: try to change gate limits in dark processing to reproduce bad pixel in the middle stripe.
2023-05-12 low gain pixels
det_calib_ave_and_max -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 100000
generates max-peds for 35998 events of xpplw3319-r0293
img-XppGon-0-Epix100a-3-xpplw3319-r0293-e035998-mode-p-max.npy
use test algorithm test-2023-04-26-chuck.py
from img-XppGon-0-Epix100a-3-xpplw3319-r0293-e035998-mode-p-max.npy
find median and +/- 6 * (median of difference)
max-peds: shape:(704, 768) size:540672 dtype:float32 [135.05005 135.3999 177.94995 180.53003 176.6001 ...]
Summary of the bad pixel status evaluation
0o1: 10021 / 540672 ( 1.853%) pixels max-peds <= 127.788
0o2: 3689 / 540672 ( 0.682%) pixels max-peds >= 203.811
saved arr_status_med-peds.npy
arr_status_med-peds.txt
cp arr_status_med-peds.txt /cds/data/psdm/xpp/xpplw3319/calib/Epix100a::CalibV1/XppGon.0:Epix100a.3/status_extra/0-end.data
see Bad Pixel Status
REFERENCES
- Paper in JAC: Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography,
- Journal of Applied Crystallography - 2022 - Sadri - Automatic bad‐pixel mask maker for X‐ray pixel detectors with.pdf
- Bad Pixel Status
- No labels
Overview
Content Tools