- Q: What can we do to improve bad pixel mask?
A:
- their code is terrible
- re-implement and try their algorithms
- Q: need help in analysis with the bad pixel mask to estimate its performance
- hit rate
- indexing efficiency
A: - Chuck's team can do that
- Chuck: start with "flat field" sample for epix_alc3 and try to find bad pixels "next to central row and "scratch"

2023-03-09 Implementation of Feature 1 and 6 for data

Data sample and detector for test

dsname exp=xpplw3319:run=293
XppGon.0:Epix100a.3

Script and accumulation of data block

Code Block

title	logfile in the beginning of the script
collapse	true

(ana-4.0.48-py3) [dubrovin@psanagpu111:~/LCLS/con-py3]$ Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
[I] det_raw_pixel_status L0146 log file: work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt
[I] UtilsCalib.py L0327 record at start: 
2023-03-09T12:11:09 user:dubrovin@psanagpu111 cwd:/cds/home/d/dubrovin/LCLS/con-py3 rel:ana-4.0.48-py3 dirrepo:work logfile:work/logs/2023/2023-03-09T121109_log_det_raw_pixel_status_dubrovin.txt command:Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
saved in: work/2023/2023_lcls1_det_raw_pixel_status.txt
[I] det_raw_pixel_status L0150 Command: Detector/app/det_raw_pixel_status -o work -n 1000 --nrecs 1000 --snrmax 8 --shwind 15,20
  Argument list: Namespace(dsname='exp=xpplw3319:run=293', source='epix_alc3', events=1000, evskip=0, steps=None, stskip=None, plotim=0, evcode=None, segind=None, dirrepo='work', nrecs=1000, logmode='INFO', dirmode=1533, filemode=436, group='ps-users', upload=False, slice='0:,0:', shwind='15,20', snrmax=8.0, databits=16383, gainbits=16383)
  Optional parameters:
    <key>      <value>              <default>
    dsname     exp=xpplw3319:run=293 exp=xpplw3319:run=293
    source     epix_alc3            epix_alc3           
    events     1000                 100                 
    evskip     0                    0                   
    steps      None                 None                
    stskip     None                 None                
    plotim     0                    0                   
    evcode     None                 None                
    segind     None                 None                
    dirrepo    work                 work                
    nrecs      1000                 1000                
    logmode    INFO                 INFO                
    dirmode    0o2775               0o2775              
    filemode   0o664                0o664               
    group      ps-users             ps-users            
    upload     False                False               
    slice      0:,0:                0:,0:               
    shwind     15,20                15,15               
    snrmax     8.0                  8.0                 
    databits   16383                16383               
    gainbits   16383                16383               

[I] UtilsRawPixelStatus.py L0351 detector type: epix100a id: 3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsRawPixelStatus.py L0361 ==== run 293
[I] UtilsRawPixelStatus.py L0367   == calibcycle 01 ==
[I] UtilsRawPixelStatus.py L0393 Ev:0001 rec:-001 raw shape:(704, 768) size:540672 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0181 created empty data block shape:(1000, 704, 768) size:540672000 dtype:uint16 [0 0 0 0 0...]
[I] UtilsRawPixelStatus.py L0393 Ev:0002 rec:0000 raw shape:(704, 768) size:540672 dtype:uint16 [3622 3968 3818 3917 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0003 rec:0001 raw shape:(704, 768) size:540672 dtype:uint16 [3629 3972 3815 3914 3689...]
[I] UtilsRawPixelStatus.py L0393 Ev:0004 rec:0002 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3970 3819 3918 3688...]
[I] UtilsRawPixelStatus.py L0393 Ev:0005 rec:0003 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3967 3821 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0011 rec:0009 raw shape:(704, 768) size:540672 dtype:uint16 [3625 3970 3819 3916 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0021 rec:0019 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3819 3916 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0031 rec:0029 raw shape:(704, 768) size:540672 dtype:uint16 [3621 3969 3820 3919 3690...]
[I] UtilsRawPixelStatus.py L0393 Ev:0041 rec:0039 raw shape:(704, 768) size:540672 dtype:uint16 [3627 3971 3817 3920 3692...]
[I] UtilsRawPixelStatus.py L0393 Ev:0101 rec:0099 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3969 3818 3914 3694...]
[I] UtilsRawPixelStatus.py L0393 Ev:0201 rec:0199 raw shape:(704, 768) size:540672 dtype:uint16 [3630 3970 3814 3918 3691...]
[I] UtilsRawPixelStatus.py L0393 Ev:0301 rec:0299 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3973 3813 3910 3693...]
[I] UtilsRawPixelStatus.py L0393 Ev:0401 rec:0399 raw shape:(704, 768) size:540672 dtype:uint16 [3624 3977 3821 3918 3695...]
[I] UtilsRawPixelStatus.py L0393 Ev:1001 rec:0999 raw shape:(704, 768) size:540672 dtype:uint16 [3626 3974 3822 3922 3694...]
[I] UtilsRawPixelStatus.py L0305 nevt:1001 accumulated requested number of records --nrecs: 1000 - break
[I] UtilsRawPixelStatus.py L0404 BREAK EVENTS
[I] UtilsRawPixelStatus.py L0414 BREAK STEPS
[I] UtilsRawPixelStatus.py L0418 BREAK RUNS
[I] UtilsRawPixelStatus.py L0246 
...

Issue for Feature 1

Image Added

Code Block

title	Logfile for Feat.1
collapse	true

Summary for data block: shape:(1000, 704, 768) size:540672000 dtype:uint16 [3623 3970 3815 3913 3691...]
[I] UtilsRawPixelStatus.py L0195 
  block record intensity MEDIAN IN FRAME: shape:(1000,) size:1000 dtype:float64
[3982. 3983. 3981. 3982. 3982. 3981. 3981. 3982. 3983. 3983. 3982. 3981.
 3982. 3983. 3982. 3982. 3982. 3981. 3983. 3981....]
[W] UtilsRawPixelStatus.py L0099 MEDIAN OF SPREAD FOR INT VALUES IS 0 replaced with STD/4 =  0.132
[I] UtilsRawPixelStatus.py L0114 ====================
    intensity_est shape:(1000,) size:1000 dtype:float64 [...]
         0 / 1000 ( 0.000%) pixels intensity_est <= 0.000
         0 / 1000 ( 0.000%) pixels intensity_est >= 16383.000
             intensity_est selected 1000 of 1000 pixels in range (0, 16383) med: 3982.000 spr: 0.132
             re-defined range for med ± 8.0*spr: (3980.946, 3983.054)
         0 / 1000 ( 0.000%) pixels intensity_est <= 3980.946
         2 / 1000 ( 0.200%) pixels intensity_est >= 3983.054
[I] UtilsRawPixelStatus.py L0199 Total number of good events: 998
[I] UtilsRawPixelStatus.py L0253 arr1_good_frames: shape:(1000,) size:1000 dtype:uint64 [1 1 1 1 1...]
[I] UtilsRawPixelStatus.py L0255 inds_good_frames: shape:(998,) size:998 dtype:int64 [...]
[  0   1   2   3   4   5   6   7   8   9  10  11  12  13  14  15  16  17
  18  19  20  21  22  23  24  25  26  27  28  29  30  31  32  33  34  35
  36  37  38  39  40  41  42  43  44  45  46  47  48  49  50  51  52  53
  54  55  56  57  58  59  60  61  62  63  64  65  66  67  68  69  70  71
  72  73  74  75  76  77  78  79  80  81  82  83  84  85  86  87  88  89
  90  91  92  93  94  95  96  98  99 100 101 102 103 104 105 106 107 108
...

median of frame intensity is int. Variation between frames is very small ~ +/- 1 ADU
median spread is also int 0
can't set med +/- 8 spr limits... does not work if implemented as described in Article
-> if spread is 0 it is evaluated as std

Feature 6

Image Added

Test of the fit to plane

Image Added

Code Block

title	logfile of the Feat.6 processing
collapse	true

...

[I] UtilsRawPixelStatus.py L0240 frame: 0991 res_med: 1.282 res_spr: 110.094 frame residuals shape:(704, 768) size:540672 dtype:float64 [-236.87714286  103.30887218  -49.50511278...]
[I] UtilsRawPixelStatus.py L0240 frame: 0992 res_med: 1.294 res_spr: 110.065 frame residuals shape:(704, 768) size:540672 dtype:float64 [-235.11755952  106.95056078  -54.98131892...]
[I] UtilsRawPixelStatus.py L0240 frame: 0993 res_med: 1.253 res_spr: 110.069 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.62863095  108.57688283  -45.21760338...]
[I] UtilsRawPixelStatus.py L0240 frame: 0994 res_med: 1.267 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-238.30857143  107.88390977  -47.92360902...]
[I] UtilsRawPixelStatus.py L0240 frame: 0995 res_med: 1.218 res_spr: 110.053 frame residuals shape:(704, 768) size:540672 dtype:float64 [-239.62184524  108.59725251  -37.18364975...]
[I] UtilsRawPixelStatus.py L0240 frame: 0996 res_med: 1.251 res_spr: 110.064 frame residuals shape:(704, 768) size:540672 dtype:float64 [-237.58547619  109.59983709  -43.21484962...]
[I] UtilsRawPixelStatus.py L0240 frame: 0997 res_med: 1.307 res_spr: 110.067 frame residuals shape:(704, 768) size:540672 dtype:float64 [-240.9114881   104.21362469  -50.66126253...]
[I] UtilsRawPixelStatus.py L0240 frame: 0998 res_med: 1.253 res_spr: 110.062 frame residuals shape:(704, 768) size:540672 dtype:float64 [-233.00535714  106.10161028  -47.79142231...]
[I] UtilsRawPixelStatus.py L0240 frame: 0999 res_med: 1.289 res_spr: 110.099 frame residuals shape:(704, 768) size:540672 dtype:float64 [-241.53529762  109.61091792  -48.24286654...]
[I] UtilsRawPixelStatus.py L0265 block of residuals: shape:(998, 704, 768) size:539590656 dtype:float64
[-239.3222619   106.86876566  -48.94020677   48.2508208  -174.55815163
  111.63287594 -128.17609649  -21.98506892  -17.79404135   80.39698622
  -86.41198622   61.77904135 -116.02993108  118.16109649 -199.64787594
   46.54315163 -238.2658208    95.92520677 -148.88376566   16.3072619 ...]
[I] UtilsRawPixelStatus.py L0266 median over frames per-pixel residuals: shape:(704, 768) size:540672 dtype:float64
[-237.54059524  107.78383145  -46.79742951   51.89206767 -174.04665414
  109.26853227 -124.40214912  -22.8108537   -14.74468515   80.47136278
  -87.92196429   64.17849154 -117.18580984  119.22582237 -198.74469768
   49.81222744 -235.53807174   93.36754543 -146.50276942   16.18511905...]
[I] UtilsRawPixelStatus.py L0267 median over frames per-pixel spread of res: shape:(704, 768) size:540672 dtype:float64
[1.91488095 1.90238878 1.80436404 1.79662124 1.58172462 1.79079887
 1.86366228 1.83122337 1.75326128 1.94361685 1.74409305 1.90914474
 1.7539317  1.96598684 1.92817356 2.05658521 1.84223058 2.07546366
 1.7810354  1.90964286...]

[I] UtilsRawPixelStatus.py L0114 ====================
    Feat.6 res_med shape:(704, 768) size:540672 dtype:float64 [...]
         0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -16383.000
         0 / 540672 ( 0.000%) pixels Feat.6 res_med >= 16383.000
             Feat.6 res_med selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.225 spr: 110.081
             re-defined range for med ± 8.0*spr: (-879.424, 881.874)
         0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
        33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
[I] UtilsRawPixelStatus.py L0114 ====================
    Feat.6 res_spr shape:(704, 768) size:540672 dtype:float64 [...]
         0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= -16383.000
         0 / 540672 ( 0.000%) pixels Feat.6 res_spr >= 16383.000
             Feat.6 res_spr selected 540672 of 540672 pixels in range (-16383, 16383) med: 1.788 spr: 0.105
             re-defined range for med ± 8.0*spr: (0.950, 2.626)
         0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
      4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
[I] UtilsRawPixelStatus.py L0284 

Summary of the bad pixel status evaluation, pixel_status_extra array
                   0o1:        0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -879.424
                   0o2:       33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 881.874
                   0o4:        0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.950
                  0o10:     4877 / 540672 ( 0.902%) pixels Feat.6 res_spr >= 2.626
    Any bad status bit:     4882 / 540672 ( 0.903%) pixels
[I] UtilsCalib.py L0150 saved:  pixel_status_extra.dat
[I] UtilsRawPixelStatus.py L0428 Consumed time 1831.462 sec

Pixel status after Feature 6

Image Added

REFERENCES

...

Versions Compared

Old Version 12

New Version 13

Key

2023-03-09 Implementation of Feature 1 and 6 for data

Data sample and detector for test

Script and accumulation of data block

Issue for Feature 1

Feature 6

Test of the fit to plane

Pixel status after Feature 6

REFERENCES

Page History

Versions Compared

Old Version 12

New Version 13

Key

2023-03-09 Implementation of Feature 1 and 6 for data

Data sample and detector for test

Script and accumulation of data block

Issue for Feature 1

Feature 6

Test of the fit to plane

Pixel status after Feature 6

REFERENCES