Page History
...
epix_alc2
epix_alc3
epix_alc4
Interim Summary
- In alc1 two data banks look different from other banks, but they are not so bad to discard them by changing thresholds.
- The "scratch" in alc3 is not seen in dark.
- The number of bad/discarded pixels can be changed by variation of thresholds.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
[I] det_raw_pixel_status L0146 log file: work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt
[I] UtilsCalib.py L0327 record at start:
2023-03-09T12:11:09 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt command:Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
saved in: work/2023/2023_lcls1_det_raw_pixel_status.txt
[I] det_raw_pixel_status L0150 Command: Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
Argument list: Namespace(dsname='exp=xpplw3319:run=293', source='epix_alc3', events=1000, evskip=0, steps=None, stskip=None, plotim=0, evcode=None, segind=None, dirrepo='work', nrecs=1000, logmode='INFO', dirmode=1533, filemode=436, group='ps-users', upload=False, slice='0:,0:', shwind='15,20', snrmax=8.0, databits=16383, gainbits=16383)
Optional parameters:
<key> <value> <default>
dsname exp=xpplw3319:run=293 exp=xpplw3319:run=293
source epix_alc3 epix_alc3
events 1000 100
evskip 0 0
steps None None
stskip None None
plotim 0 0
evcode None None
segind None None
dirrepo work work
nrecs 1000 1000
logmode INFO INFO
dirmode 0o2775 0o2775
filemode 0o664 0o664
group ps-users ps-users
upload False False
slice 0:,0: 0:,0:
shwind 15,20 15,15
snrmax 8.0 8.0
databits 16383 16383
gainbits 0 16383 16383 0
[I] UtilsRawPixelStatus.py L0351 detector type: epix100a id: 3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsRawPixelStatus.py L0361 ==== run 293
[I] UtilsRawPixelStatus.py L0367 == calibcycle 01 ==
[I] UtilsRawPixelStatus.py L0393 Ev:0001 rec:-001 raw shape:(704, 768) size:540672 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0181 created empty data block shape:(1000, 704, 768) size:540672000 dtype:uint16 [0 0 0 0 0...]
[I] UtilsRawPixelStatus.py L0393 Ev:0002 rec:0000 raw shape:(704, 768) size:540672 dtype:uint16 [3622 3968 3818 3917 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0003 rec:0001 raw shape:(704, 768) size:540672 dtype:uint16 [3629 3972 3815 3914 3689...]
[I] UtilsRawPixelStatus.py L0393 Ev:0004 rec:0002 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3970 3819 3918 3688...]
[I] UtilsRawPixelStatus.py L0393 Ev:0005 rec:0003 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3967 3821 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0011 rec:0009 raw shape:(704, 768) size:540672 dtype:uint16 [3625 3970 3819 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0021 rec:0019 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3819 3916 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0031 rec:0029 raw shape:(704, 768) size:540672 dtype:uint16 [3621 3969 3820 3919 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0041 rec:0039 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3817 3920 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0101 rec:0099 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3969 3818 3914 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0201 rec:0199 raw shape:(704, 768) size:540672 dtype:uint16 [3630 3970 3814 3918 3691...]
[I] UtilsRawPixelStatus.py L0393 Ev:0301 rec:0299 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3973 3813 3910 3693...]
[I] UtilsRawPixelStatus.py L0393 Ev:0401 rec:0399 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3977 3821 3918 3695...]
[I] UtilsRawPixelStatus.py L0393 Ev:1001 rec:0999 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3974 3822 3922 3694...]
[I] UtilsRawPixelStatus.py L0305 nevt:1001 accumulated requested number of records --nrecs: 1000 - break
[I] UtilsRawPixelStatus.py L0404 BREAK EVENTS
[I] UtilsRawPixelStatus.py L0414 BREAK STEPS
[I] UtilsRawPixelStatus.py L0418 BREAK RUNS
[I] UtilsRawPixelStatus.py L0246
... |
Issue for Feature 1
| Code Block | ||||
|---|---|---|---|---|
| ||||
Summary for data block: shape:(1000, 704, 768) size:540672000 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0195
block record intensity MEDIAN IN FRAME: shape:(1000,) size:1000 dtype:float64
[3982. 3983. 3981. 3982. 3982. 3981. 3981. 3982. 3983. 3983. 3982. 3981.
3982. 3983. 3982. 3982. 3982. 3981. 3983. 3981....]
[W] UtilsRawPixelStatus.py L0099 MEDIAN OF SPREAD FOR INT VALUES IS 0 replaced with STD/4 = 0.132
[I] UtilsRawPixelStatus.py L0114 ====================
intensity_est shape:(1000,) size:1000 dtype:float64 [...]
0 / 1000 ( 0.000%) pixels intensity_est <= 0.000
0 / 1000 ( 0.000%) pixels intensity_est >= 16383.000
intensity_est selected 1000 of 1000 pixels in range (0, 16383) med: 3982.000 spr: 0.132
re-defined range for med ± 8.0*spr: (3980.946, 3983.054)
0 / 1000 ( 0.000%) pixels intensity_est <= 3980.946
2 / 1000 ( 0.200%) pixels intensity_est >= 3983.054
[I] UtilsRawPixelStatus.py L0199 Total number of good events: 998
[I] UtilsRawPixelStatus.py L0253 arr1_good_frames: shape:(1000,) size:1000 dtype:uint64 [1 1 1 1 1...]
[I] UtilsRawPixelStatus.py L0255 inds_good_frames: shape:(998,) size:998 dtype:int64 [...]
[ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71
72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89
90 91 92 93 94 95 96 98 99 100 101 102 103 104 105 106 107 108
... |
- median of frame intensity is int. Variation between frames is very small ~ +/- 1 ADU
- median spread is also int 0
- can't set med +/- 8 spr limits... does not work if implemented as described in Article
-> if spread is 0 it is evaluated as std
Feature 2 and 3 for dark run
Implemented on 2023-04-06 exactly as described. Median is used for robust statistics parameters evaluation.
Feature 4 and 5 for gain modes
TBD
Feature 6
Test of the fit to plane
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
...
[I] UtilsRawPixelStatus.py L0240 frame: 0991 res_med: 1.282 res_spr: 110.094 frame residuals shape:(704, 768) size:540672 dtype:float64 [-236.87714286 103.30887218 -49.50511278...]
[I] UtilsRawPixelStatus.py L0240 frame: 0992 res_med: 1.294 res_spr: 110.065 frame residuals shape:(704, 768) size:540672 dtype:float64 [-235.11755952 106.95056078 -54.98131892...]
[I] UtilsRawPixelStatus.py L0240 frame: 0993 res_med: 1.253 res_spr: 110.069 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.62863095 108.57688283 -45.21760338...]
[I] UtilsRawPixelStatus.py L0240 frame: 0994 res_med: 1.267 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-238.30857143 107.88390977 -47.92360902...]
[I] UtilsRawPixelStatus.py L0240 frame: 0995 res_med: 1.218 res_spr: 110.053 frame residuals shape:(704, 768) size:540672 dtype:float64 [-239.62184524 108.59725251 -37.18364975...]
[I] UtilsRawPixelStatus.py L0240 frame: 0996 res_med: 1.251 res_spr: 110.064 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.58547619 109.59983709 -43.21484962...]
[I] UtilsRawPixelStatus.py L0240 frame: 0997 res_med: 1.307 res_spr: 110.067 frame residuals shape:(704, 768) size:540672 dtype:float64 [-240.9114881 104.21362469 -50.66126253...]
[I] UtilsRawPixelStatus.py L0240 frame: 0998 res_med: 1.253 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-233.00535714 106.10161028 -47.79142231...]
[I] UtilsRawPixelStatus.py L0240 frame: 0999 res_med: 1.289 res_spr: 110.099 frame residuals shape:(704, 768) size:540672 dtype:float64 [-241.53529762 109.61091792 -48.24286654...]
[I] UtilsRawPixelStatus.py L0265 block of residuals: shape:(998, 704, 768) size:539590656 dtype:float64
[-239.3222619 106.86876566 -48.94020677 48.2508208 -174.55815163
111.63287594 -128.17609649 -21.98506892 -17.79404135 80.39698622
-86.41198622 61.77904135 -116.02993108 118.16109649 -199.64787594
46.54315163 -238.2658208 95.92520677 -148.88376566 16.3072619 ...]
[I] UtilsRawPixelStatus.py L0266 median over frames per-pixel residuals: shape:(704, 768) size:540672 dtype:float64
[-237.54059524 107.78383145 -46.79742951 51.89206767 -174.04665414
109.26853227 -124.40214912 -22.8108537 -14.74468515 80.47136278
-87.92196429 64.17849154 -117.18580984 119.22582237 -198.74469768
49.81222744 -235.53807174 93.36754543 -146.50276942 16.18511905...]
[I] UtilsRawPixelStatus.py L0267 median over frames per-pixel spread of res: shape:(704, 768) size:540672 dtype:float64
[1.91488095 1.90238878 1.80436404 1.79662124 1.58172462 1.79079887
1.86366228 1.83122337 1.75326128 1.94361685 1.74409305 1.90914474
1.7539317 1.96598684 1.92817356 2.05658521 1.84223058 2.07546366
1.7810354 1.90964286...]
[I] UtilsRawPixelStatus.py L0114 ====================
Feat.6 res_med shape:(704, 768) size:540672 dtype:float64 [...]
0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -16383.000
0 / 540672 ( 0.000%) pixels Feat.6 res_med >= 16383.000
Feat.6 res_med selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.225 spr: 110.081
re-defined range for med ± 8.0*spr: (-879.424, 881.874)
0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
[I] UtilsRawPixelStatus.py L0114 ====================
Feat.6 res_spr shape:(704, 768) size:540672 dtype:float64 [...]
0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= -16383.000
0 / 540672 ( 0.000%) pixels Feat.6 res_spr >= 16383.000
Feat.6 res_spr selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.788 spr: 0.105
re-defined range for med ± 8.0*spr: (0.950, 2.626)
0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
[I] UtilsRawPixelStatus.py L0284
Summary of the bad pixel status evaluation, pixel_status_extra array
0o1: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
0o2: 33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
0o4: 0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
0o10: 4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
Any bad status bit: 4882 / 540672 ( 0.903%) pixels
[I] UtilsCalib.py L0150 saved: pixel_status_extra.dat
[I] UtilsRawPixelStatus.py L0428 Consumed time 1831.462 sec |
...
- Feat. 6 adds more bad pixels, especially stripe in the lower right ASIC
- "scratch" seen in data is not reproduced
REFERENCES
Chuck wants to find "scratch" automatically
2023-04-06 Feature 2 and 3 for dark data vs our standard dark processing
- dark data set: exp=xpplw3319:run=287
- detector: XppGon.0:Epix100a.3, alias: epix_alc3
- This data set contains ~1200 events, of which initial 1000 is used in processing.
- By default features 1,2,3 are used. Feature 4 and 5 for gain factors are not implemented and are irrelevant to epix100a.
- Feature 1 - good frame selection discards 1/0/0 frames for SNR=3/5/8, respectively
| Code Block | ||||
|---|---|---|---|---|
| ||||
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R3 --gmode SNR3
Summary of the bad pixel status evaluation for SNR = 3.00, status_data array
0o1: 6445 / 540672 ( 1.192%) pixels Feat.2 mean <= 3588.608
0o2: 13484 / 540672 ( 2.494%) pixels Feat.2 mean >= 4374.797
0o4: 5600 / 540672 ( 1.036%) pixels Feat.3 std <= 2.314
0o10: 34098 / 540672 ( 6.307%) pixels Feat.3 std >= 3.263
Any bad status bit: 57409 / 540672 (10.618%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R5 --gmode SNR5
Summary of the bad pixel status evaluation for SNR=5.00, status_data array
0o1: 5 / 540672 ( 0.001%) pixels Feat.2 mean <= 3326.536
0o2: 87 / 540672 ( 0.016%) pixels Feat.2 mean >= 4636.866
0o4: 878 / 540672 ( 0.162%) pixels Feat.3 std <= 1.999
0o10: 12739 / 540672 ( 2.356%) pixels Feat.3 std >= 3.579
Any bad status bit: 13675 / 540672 ( 2.529%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -R8 --gmode SNR8
Summary of the bad pixel status evaluation for SNR=8.00, status_data array
0o1: 0 / 540672 ( 0.000%) pixels Feat.2 mean <= 2933.437
0o2: 33 / 540672 ( 0.006%) pixels Feat.2 mean >= 5029.965
0o4: 0 / 540672 ( 0.000%) pixels Feat.3 std <= 1.525
0o10: 5314 / 540672 ( 0.983%) pixels Feat.3 std >= 4.053
Any bad status bit: 5319 / 540672 ( 0.984%) pixels |
| Code Block | ||||
|---|---|---|---|---|
| ||||
det_ndarr_raw_proc -d exp=xpplw3319:run=287 -s XppGon.0:Epix100a.3 -n 1000 -o work
raw data found/selected in 1000 events
[I] UtilsCalib.py L0683 begin data summary stage
[I] UtilsCalib.py L0112 evaluate_limits RMS: ave=2.317 std=0.370 limits low=0.097 high=4.536
[I] UtilsCalib.py L0112 evaluate_limits AVE: ave=3983.565 std=188.830 limits low=2850.584 high=5116.546
[I] UtilsCalib.py L0721 bad pixel status:
status 1: 643 pixel rms > 4.536
status 2: 0 pixel rms < 0.097
status 4: 0 pixel intensity > 16000 in more than 0.1 fraction (99/999) of non-empty events
status 8: 0 pixel intensity < 1 in more than 0.1 fraction (99/999) of non-empty events
status 16: 31 pixel average > 5116.55
status 32: 0 pixel average < 2850.58 |
2023-04-06 Feature 6 for regular data for SNR=3 and 5
- dark data set: exp=xpplw3319:run=287
- detector: XppGon.0:Epix100a.3, alias: epix_alc3
| Code Block | ||||
|---|---|---|---|---|
| ||||
det_raw_pixel_status -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 1000 -R3 --gmode SNR3-F6 -F 1,6
Summary of the bad pixel status evaluation for SNR=3.00, status_data array
0o1: 449 / 540672 ( 0.083%) pixels Feat.6 res_med <= -330.137
0o2: 379 / 540672 ( 0.070%) pixels Feat.6 res_med >= 331.245
0o4: 8305 / 540672 ( 1.536%) pixels Feat.6 res_spr <= 1.472
0o10: 34625 / 540672 ( 6.404%) pixels Feat.6 res_spr >= 2.095
Any bad status bit: 43648 / 540672 ( 8.073%) pixels
det_raw_pixel_status -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 1000 -R5 --gmode SNR5-F6 -F 1,6
Summary of the bad pixel status evaluation for SNR=5.00, status_data array
0o1: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -550.599
0o2: 45 / 540672 ( 0.008%) pixels Feat.6 res_med >= 551.708
0o4: 1005 / 540672 ( 0.186%) pixels Feat.6 res_spr <= 1.265
0o10: 12484 / 540672 ( 2.309%) pixels Feat.6 res_spr >= 2.302
Any bad status bit: 13501 / 540672 ( 2.497%) pixels |
Interim summary
- Number of bad pixels depends on SNR parameter
- Stripe of bad pixels in the middle of the detector can be caught by Feature 3 and 6 at SNR=5.
- "Scratch" observed by Chuck is not seen detected ...
2023-04-07 mtg summary
- Chuck: try to search for scratch in ASIC ONLY!
- Chris: try to change gate limits in dark processing to reproduce bad pixel in the middle stripe.
2023-05-12 low gain pixels
det_calib_ave_and_max -d exp=xpplw3319:run=293 -s XppGon.0:Epix100a.3 -n 100000
generates max-peds for 35998 events of xpplw3319-r0293
img-XppGon-0-Epix100a-3-xpplw3319-r0293-e035998-mode-p-max.npy
use test algorithm test-2023-04-26-chuck.py
from img-XppGon-0-Epix100a-3-xpplw3319-r0293-e035998-mode-p-max.npy
find median and +/- 6 * (median of difference)
max-peds: shape:(704, 768) size:540672 dtype:float32 [135.05005 135.3999 177.94995 180.53003 176.6001 ...]
Summary of the bad pixel status evaluation
0o1: 10021 / 540672 ( 1.853%) pixels max-peds <= 127.788
0o2: 3689 / 540672 ( 0.682%) pixels max-peds >= 203.811
saved arr_status_med-peds.npy
arr_status_med-peds.txt
cp arr_status_med-peds.txt /cds/data/psdm/xpp/xpplw3319/calib/Epix100a::CalibV1/XppGon.0:Epix100a.3/status_extra/0-end.data
see Bad Pixel Status
REFERENCES
- Paper in JAC: Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography,
- Journal of Applied Crystallography - 2022 - Sadri - Automatic bad‐pixel
- Paper in JAC: Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography,
- Journal of Applied Crystallography - 2022 - Sadri - Automatic bad‐pixel mask maker for X‐ray pixel detectors with.pdf
- Bad Pixel Status
Overview
Content Tools