- Created by Mikhail Dubrovin, last modified on Jan 31, 2023
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 12 Next »
Issue
e-mail from Chuck
Yoon, Chun Hong <yoon82@slac.stanford.edu> 2023-01-24 10:13pm Hi Mikhail, I came across a user working on xpplw3319 dataset. It seemed that our LCLS mask was not sufficient for their analysis. Please see attached 4 detector averaged over multiple events and 4 detector masks (run293, epic_alc1,2,3,4). Do you think setting more conservative parameters in selecting bad pixel map would help in this case? Best, Chuck
Dataset
xpplw3319 run 293, dark run 287
datinfo -e xpplw3319 -r 293 -d epix_alc1
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.1)', alias='epix_alc1')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.2)', alias='epix_alc2')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.3)', alias='epix_alc3')
EventKey(type=psana.Epix.ElementV3, src='DetInfo(XppGon.0:Epix100a.4)', alias='epix_alc4')
[I] L0120: calibDir: /cds/data/psdm/xpp/xpplw3319/calib
Average image and mask for four detectors
Dark processing check
https://pswww.slac.stanford.edu/lgbk/lgbk/xpplw3319/#eLog
Jul/16/2021 06:11:15 287 dark - Restored
Jul/16/2021 07:44:51 293
===
Restore 287 dark
/cds/data/psdm/xpp/xpplw3319/xtc/xpplw3319-r028*
Command and logfiles for four detectors
det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000 [I] UtilsCalib.py L0327 record at start: 2023-01-26T09:13:07 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000 saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt [I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.1 -o work --nrecs1 50 --nrecs 1000 Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.1', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50) Optional parameters: <key> <value> <default> dsname exp=xpplw3319:run=287 None source XppGon.0:Epix100a.1 None ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt events 1000 1000 evskip 0 0 intlow 1 1 inthig 16000 16000 rmslow 0.001 0.001 rmshig 16000 16000 fraclm 0.1 0.1 nsigma 6.0 6.0 plotim 0 0 verbos 7 7 savebw 255 255 intnlo 6.0 6.0 intnhi 6.0 6.0 rmsnlo 6.0 6.0 rmsnhi 6.0 6.0 evcode None None dirrepo work ./work logmode INFO INFO dirmode 0o2775 0o2775 filemode 0o664 0o664 group ps-users ps-users nrecs 1000 100 nrecs1 50 50 [I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287 [I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.1 [I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...] [I] UtilsDarkProc.py L0168 Event: 0, time= 0.127 sec, dt=0.127 sec [I] UtilsDarkProc.py L0168 Event: 1, time= 0.130 sec, dt=0.003 sec [I] UtilsDarkProc.py L0168 Event: 2, time= 0.135 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 3, time= 0.139 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 4, time= 0.144 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 10, time= 0.171 sec, dt=0.027 sec [I] UtilsDarkProc.py L0168 Event: 20, time= 0.193 sec, dt=0.023 sec [I] UtilsDarkProc.py L0168 Event: 30, time= 0.214 sec, dt=0.021 sec [I] UtilsDarkProc.py L0168 Event: 40, time= 0.233 sec, dt=0.020 sec [I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [3761 3889 3838 3956 3827...] [I] UtilsCalib.py L0587 proc_block pre-processing time 2.491 sec results for median over pixels intensities: 0.500 fraction of the event spectrum is below 3920.000 ADU - pedestal estimator 0.050 fraction of the event spectrum is below 3916.000 ADU - gate low limit 0.950 fraction of the event spectrum is below 3925.000 ADU - gate upper limit event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator [I] UtilsCalib.py L0641 data block processing total time 2.514 sec arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [3758. 3885. 3839. 3955. 3828....] abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [1.5 1. 2. 2. 2. ...] gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [3754 3882 3836 3951 3823...] gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [3762 3889 3845 3960 3833...] [I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16 [I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4045 4231 4160 4173 3957...] 1st stage event block processing is completed [I] UtilsDarkProc.py L0168 Event: 100, time= 4.665 sec, dt=4.432 sec [I] UtilsDarkProc.py L0168 Event: 200, time= 6.616 sec, dt=1.952 sec [I] UtilsDarkProc.py L0168 Event: 300, time= 8.555 sec, dt=1.939 sec [I] UtilsDarkProc.py L0168 Event: 400, time= 10.518 sec, dt=1.962 sec [I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000 [I] UtilsCalib.py L0679 summary [I] UtilsCalib.py L0680 ________________________________________________________________________________ raw data found/selected in 1000 events [I] UtilsCalib.py L0683 begin data summary stage [I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.203 std=0.445 limits low=0.001 high=4.872 [I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3900.681 std=209.658 limits low=2642.732 high=5158.629 [I] UtilsCalib.py L0721 bad pixel status: status 1: 379 pixel rms > 4.872 status 2: 0 pixel rms < 0.001 status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events status 16: 15 pixel average > 5158.63 status 32: 0 pixel average < 2642.73 File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.1-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved [I] UtilsDarkProc.py L0174 ________________________________________________________________________________ Total consumed time = 22.977433 sec.
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000 [I] UtilsCalib.py L0327 record at start: 2023-01-26T09:17:49 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000 saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt [I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.2 -o work --nrecs1 50 --nrecs 1000 Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.2', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50) Optional parameters: <key> <value> <default> dsname exp=xpplw3319:run=287 None source XppGon.0:Epix100a.2 None ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt events 1000 1000 evskip 0 0 intlow 1 1 inthig 16000 16000 rmslow 0.001 0.001 rmshig 16000 16000 fraclm 0.1 0.1 nsigma 6.0 6.0 plotim 0 0 verbos 7 7 savebw 255 255 intnlo 6.0 6.0 intnhi 6.0 6.0 rmsnlo 6.0 6.0 rmsnhi 6.0 6.0 evcode None None dirrepo work ./work logmode INFO INFO dirmode 0o2775 0o2775 filemode 0o664 0o664 group ps-users ps-users nrecs 1000 100 nrecs1 50 50 [I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287 [I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.2 [I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...] [I] UtilsDarkProc.py L0168 Event: 0, time= 0.085 sec, dt=0.085 sec [I] UtilsDarkProc.py L0168 Event: 1, time= 0.089 sec, dt=0.003 sec [I] UtilsDarkProc.py L0168 Event: 2, time= 0.093 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 3, time= 0.098 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 4, time= 0.102 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 10, time= 0.128 sec, dt=0.026 sec [I] UtilsDarkProc.py L0168 Event: 20, time= 0.151 sec, dt=0.023 sec [I] UtilsDarkProc.py L0168 Event: 30, time= 0.172 sec, dt=0.021 sec [I] UtilsDarkProc.py L0168 Event: 40, time= 0.192 sec, dt=0.021 sec [I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [4154 4215 4185 4397 4179...] [I] UtilsCalib.py L0587 proc_block pre-processing time 2.500 sec results for median over pixels intensities: 0.500 fraction of the event spectrum is below 4072.000 ADU - pedestal estimator 0.050 fraction of the event spectrum is below 4068.000 ADU - gate low limit 0.950 fraction of the event spectrum is below 4076.000 ADU - gate upper limit event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator [I] UtilsCalib.py L0641 data block processing total time 2.524 sec arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [4156. 4217. 4181. 4397. 4181....] abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [2. 2. 2. 2. 2....] gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [4151 4213 4178 4393 4175...] gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [4159 4222 4186 4400 4184...] [I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16 [I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4300 4651 4343 4538 4177...] 1st stage event block processing is completed [I] UtilsDarkProc.py L0168 Event: 100, time= 4.653 sec, dt=4.461 sec [I] UtilsDarkProc.py L0168 Event: 200, time= 6.686 sec, dt=2.033 sec [I] UtilsDarkProc.py L0168 Event: 300, time= 8.654 sec, dt=1.969 sec [I] UtilsDarkProc.py L0168 Event: 400, time= 10.682 sec, dt=2.028 sec [I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000 [I] UtilsCalib.py L0679 summary [I] UtilsCalib.py L0680 ________________________________________________________________________________ raw data found/selected in 1000 events [I] UtilsCalib.py L0683 begin data summary stage [I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.330 std=0.995 limits low=0.001 high=8.303 [I] UtilsCalib.py L0112 evaluate_limits AVE: ave=4065.543 std=226.214 limits low=2708.260 high=5422.827 [I] UtilsCalib.py L0721 bad pixel status: status 1: 239 pixel rms > 8.303 status 2: 0 pixel rms < 0.001 status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events status 16: 26 pixel average > 5422.83 status 32: 1 pixel average < 2708.26 File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.2-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved [I] UtilsDarkProc.py L0174 ________________________________________________________________________________ Total consumed time = 23.291649 sec. [I] det_ndarr_raw_proc L0141 log file: None (ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000 [I] UtilsCalib.py L0327 record at start: 2023-01-26T09:21:06 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000 saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt [I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -o work --nrecs1 50 --nrecs 1000 Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.3', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50) Optional parameters: <key> <value> <default> dsname exp=xpplw3319:run=287 None source XppGon.0:Epix100a.3 None ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt events 1000 1000 evskip 0 0 intlow 1 1 inthig 16000 16000 rmslow 0.001 0.001 rmshig 16000 16000 fraclm 0.1 0.1 nsigma 6.0 6.0 plotim 0 0 verbos 7 7 savebw 255 255 intnlo 6.0 6.0 intnhi 6.0 6.0 rmsnlo 6.0 6.0 rmsnhi 6.0 6.0 evcode None None dirrepo work ./work logmode INFO INFO dirmode 0o2775 0o2775 filemode 0o664 0o664 group ps-users ps-users nrecs 1000 100 nrecs1 50 50 [I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287 [I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.3 [I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...] [I] UtilsDarkProc.py L0168 Event: 0, time= 0.074 sec, dt=0.074 sec [I] UtilsDarkProc.py L0168 Event: 1, time= 0.078 sec, dt=0.003 sec [I] UtilsDarkProc.py L0168 Event: 2, time= 0.082 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 3, time= 0.087 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 4, time= 0.091 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 10, time= 0.118 sec, dt=0.027 sec [I] UtilsDarkProc.py L0168 Event: 20, time= 0.141 sec, dt=0.023 sec [I] UtilsDarkProc.py L0168 Event: 30, time= 0.161 sec, dt=0.021 sec [I] UtilsDarkProc.py L0168 Event: 40, time= 0.183 sec, dt=0.022 sec [I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [3841 3962 3836 3825 3845...] [I] UtilsCalib.py L0587 proc_block pre-processing time 2.515 sec results for median over pixels intensities: 0.500 fraction of the event spectrum is below 3982.000 ADU - pedestal estimator 0.050 fraction of the event spectrum is below 3977.000 ADU - gate low limit 0.950 fraction of the event spectrum is below 3987.000 ADU - gate upper limit event spectrum spread median(abs(raw-med)): 2.000 ADU - spectral peak width estimator [I] UtilsCalib.py L0641 data block processing total time 2.537 sec arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [3844. 3963. 3839. 3831.5 3845. ...] abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [2. 2. 2. 1.5 2. ...] gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [3837 3958 3835 3825 3840...] gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [3847 3969 3843 3836 3850...] [I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16 [I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [3632 3974 3821 3918 3696...] 1st stage event block processing is completed [I] UtilsDarkProc.py L0168 Event: 100, time= 4.701 sec, dt=4.518 sec [I] UtilsDarkProc.py L0168 Event: 200, time= 6.827 sec, dt=2.126 sec [I] UtilsDarkProc.py L0168 Event: 300, time= 8.874 sec, dt=2.047 sec [I] UtilsDarkProc.py L0168 Event: 400, time= 10.912 sec, dt=2.038 sec [I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000 [I] UtilsCalib.py L0679 summary [I] UtilsCalib.py L0680 ________________________________________________________________________________ raw data found/selected in 1000 events [I] UtilsCalib.py L0683 begin data summary stage [I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.317 std=0.370 limits low=0.097 high=4.536 [I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3983.565 std=188.830 limits low=2850.584 high=5116.546 [I] UtilsCalib.py L0721 bad pixel status: status 1: 643 pixel rms > 4.536 status 2: 0 pixel rms < 0.097 status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events status 16: 31 pixel average > 5116.55 status 32: 0 pixel average < 2850.58 File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.3-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved [I] UtilsDarkProc.py L0174 ________________________________________________________________________________ Total consumed time = 23.812071 sec. [I] det_ndarr_raw_proc L0141 log file: None (ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000 [I] UtilsCalib.py L0327 record at start: 2023-01-26T09:22:44 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:None command:/cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000 saved in: /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_ndarr_raw_proc.txt [I] det_ndarr_raw_proc L0135 Command: /cds/home/d/dubrovin/LCLS/con-py3/arch/x86_64-rhel7-gcc48-opt/bin/det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.4 -o work --nrecs1 50 --nrecs 1000 Argument list: Namespace(dsname='exp=xpplw3319:run=287', source='XppGon.0:Epix100a.4', ofname='nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt', events=1000, evskip=0, intlow=1, inthig=16000, rmslow=0.001, rmshig=16000, fraclm=0.1, nsigma=6.0, plotim=0, verbos=7, savebw=255, intnlo=6.0, intnhi=6.0, rmsnlo=6.0, rmsnhi=6.0, evcode=None, dirrepo='work', logmode='INFO', dirmode=1533, filemode=436, group='ps-users', nrecs=1000, nrecs1=50) Optional parameters: <key> <value> <default> dsname exp=xpplw3319:run=287 None source XppGon.0:Epix100a.4 None ofname nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt nda-#exp-#run-#src-#evts-#type-#date-#time-#fid-#sec-#nsec.txt events 1000 1000 evskip 0 0 intlow 1 1 inthig 16000 16000 rmslow 0.001 0.001 rmshig 16000 16000 fraclm 0.1 0.1 nsigma 6.0 6.0 plotim 0 0 verbos 7 7 savebw 255 255 intnlo 6.0 6.0 intnhi 6.0 6.0 rmsnlo 6.0 6.0 rmsnhi 6.0 6.0 evcode None None dirrepo work ./work logmode INFO INFO dirmode 0o2775 0o2775 filemode 0o664 0o664 group ps-users ps-users nrecs 1000 100 nrecs1 50 50 [I] UtilsDarkProc.py L0129 Raw data processing of dataset: exp=xpplw3319:run=287 [I] UtilsDarkProc.py L0065 create DarkProcDet object for XppGon.0:Epix100a.4 [I] UtilsCalib.py L0793 created empty data block shape:(50, 704, 768) size:27033600 dtype:uint16 [0 0 0 0 0...] [I] UtilsDarkProc.py L0168 Event: 0, time= 0.074 sec, dt=0.074 sec [I] UtilsDarkProc.py L0168 Event: 1, time= 0.078 sec, dt=0.003 sec [I] UtilsDarkProc.py L0168 Event: 2, time= 0.082 sec, dt=0.004 sec [I] UtilsDarkProc.py L0168 Event: 3, time= 0.086 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 4, time= 0.091 sec, dt=0.005 sec [I] UtilsDarkProc.py L0168 Event: 10, time= 0.118 sec, dt=0.027 sec [I] UtilsDarkProc.py L0168 Event: 20, time= 0.141 sec, dt=0.023 sec [I] UtilsDarkProc.py L0168 Event: 30, time= 0.161 sec, dt=0.021 sec [I] UtilsDarkProc.py L0168 Event: 40, time= 0.181 sec, dt=0.020 sec [I] UtilsCalib.py L0536 begin pricessing of the data block shape:(50, 704, 768) size:27033600 dtype:uint16 [4523 4558 4216 4521 4508...] [I] UtilsCalib.py L0587 proc_block pre-processing time 2.555 sec results for median over pixels intensities: 0.500 fraction of the event spectrum is below 4577.000 ADU - pedestal estimator 0.050 fraction of the event spectrum is below 4571.000 ADU - gate low limit 0.950 fraction of the event spectrum is below 4583.000 ADU - gate upper limit event spectrum spread median(abs(raw-med)): 2.500 ADU - spectral peak width estimator [I] UtilsCalib.py L0641 data block processing total time 2.577 sec arr_med[100:105] shape:(704, 768) size:540672 dtype:float64 [4521. 4554. 4220. 4524.5 4508.5...] abs_dev[100:105] shape:(704, 768) size:540672 dtype:float64 [1.5 2. 2. 2.5 2.5...] gate_lo[100:105] shape:(704, 768) size:540672 dtype:uint16 [4516 4550 4215 4519 4505...] gate_hi[100:105] shape:(704, 768) size:540672 dtype:uint16 [4525 4563 4224 4529 4513...] [I] UtilsCalib.py L0653 Stage 2 initialization for raw shape (704, 768) and dtype uint16 [I] UtilsCalib.py L0778 add to gated average statistics the block of initial data shape:(50, 704, 768) size:27033600 dtype:uint16 [4626 4797 4659 4839 4565...] 1st stage event block processing is completed [I] UtilsDarkProc.py L0168 Event: 100, time= 4.722 sec, dt=4.541 sec [I] UtilsDarkProc.py L0168 Event: 200, time= 6.681 sec, dt=1.959 sec [I] UtilsDarkProc.py L0168 Event: 300, time= 8.618 sec, dt=1.937 sec [I] UtilsDarkProc.py L0168 Event: 400, time= 10.753 sec, dt=2.135 sec [I] UtilsCalib.py L0811 record 999 event loop is terminated, --nrecs=1000 [I] UtilsCalib.py L0679 summary [I] UtilsCalib.py L0680 ________________________________________________________________________________ raw data found/selected in 1000 events [I] UtilsCalib.py L0683 begin data summary stage [I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.965 std=0.396 limits low=0.587 high=5.344 [I] UtilsCalib.py L0112 evaluate_limits AVE: ave=4581.344 std=278.259 limits low=2911.793 high=6250.895 [I] UtilsCalib.py L0721 bad pixel status: status 1: 286 pixel rms > 5.344 status 2: 1 pixel rms < 0.587 status 4: 1 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events status 16: 373 pixel average > 6250.9 status 32: 1 pixel average < 2911.79 File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-ave-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-rms-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-sta-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-msk-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-max-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved File nda-xpplw3319-r0287-XppGon.0:Epix100a.4-e000999-min-2021-07-16-06:11:15-47256-1626441075-950830948.txt is saved [I] UtilsDarkProc.py L0174 ________________________________________________________________________________ Total consumed time = 23.475768 sec. [I] det_ndarr_raw_proc L0141 log file: None (ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$
Dark average (pedestals) and pixel_rms images with spectra
epix_alc1
epix_alc2
epix_alc3
epix_alc4
Summary
- In alc1 two data banks look different from other banks, but they are not so bad to discard them by changing thresholds.
- The "scratch" in alc3 is not seen in dark.
- The number of bad/discarded pixels can be changed by variation of thresholds.
Summary of the mtg with Chuck
Discuss article Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography
- Q: definition of robust parameters
A: robust mean is median xi
robust sigma is median |xi-mu|
- Q: difference in algorithms
A:
- for dark intensity and rms algorithms are about the same
- Philip evaluated gains for a few detectors
- but in general we do not have/use bad pixel status for gain factors (generated from flat field)
WE NEED IN LIGHT FLAT FIELD DATA IN ALL GAIN RANGES ALONG WITH DARK
- for gain and bad pixel status
- Q: What can we do to improve bad pixel mask?
A:
- their code is terrible
- re-implement and try their algorithms
- Q: need help in analysis with the bad pixel mask to estimate its performance
- hit rate
- indexing efficiency
A: - Chuck's team can do that
- Chuck: start with "flat field" sample for epix_alc3 and try to find bad pixels "next to central row and "scratch"
REFERENCES
- Paper in JAC: Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography,
- Journal of Applied Crystallography - 2022 - Sadri - Automatic bad‐pixel mask maker for X‐ray pixel detectors with.pdf
- No labels