Content
Command examples
Script and its features
- Script
det_pixel_status processes dark or light data and produces 2-d image or per-panel pixel status files. - Implemented algorithms, named here as Features after Ref.2,3, are developed in R&D with Chuck briefly documented in Ref.1.
- Tested for epix100a - other detectors not tested...
All algorithms/features are wrapped under the common hood of event processing loop and files' I/O. But, not all algorithms/features can be applied simultaneously due to different data types (dark or light) and different processing algorithms. Suggested combinations for -F parameters are shown in three examples below features with associated type of generated status constants in repository
- -F 1,2,3 # status_dark
- -F 1,6 # status_light
- -F 11 # status_max
Location of repository and calibration files can be found in the log file, see also Ref.5.
Get help
ana-4.0.51-py3 [username@sdfiana001:~/]$ det_pixel_status -h
usage:
det_pixel_status works with 2-d raw arrays ONLY! For 3-d the segment index --segind needs to be specified
det_pixel_status -k <dataset> -d <detname> ...
Ex.1: det_pixel_status -k exp=xpplw3319:run=287 -d XppGon.0:Epix100a.3 -F 1,2,3 -n 1000 # dark processing for Feat.1 & 2 & 3
Ex.2: det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 1,6 -n 1000 # light processing for Feat.1 & 6
Ex.3: det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 11, -n 1000000 # light processing for Feat.11
Feature # stands for
1: mean intensity of frames in good range
2: dark mean in good range
3: dark rms in good range
4: NOT-IMPLEMENTED for gain stage indicator values
5: NOT-IMPLEMENTED for gain stage indicator values
6: light average SNR of pixels over time
11: light intensity max-peds in good range - should be processed separately from all other features on entire/large set of events.
Help: det_pixel_status -h
Evaluates pixel status of any detector/segment using raw dark/light data
optional arguments:
-h, --help show this help message and exit
-k DSKWARGS, --dskwargs DSKWARGS
(str) DataSource parameters, default = exp=xpplw3319:run=293
-d DETNAME, --detname DETNAME
(str) detector/detname name, default = epix_alc3
-n EVENTS, --events EVENTS
(int) maximal number of events total (in runs, steps), default = 1000
-m EVSKIP, --evskip EVSKIP
(int) number of events to skip in the beginning of each step, default = 0
-N STEPS, --steps STEPS
(int) maximal number of steps or all by default, default = None
-M STSKIP, --stskip STSKIP
(int) number of steps to skip before processing or process all by default, default = None
-c EVCODE, --evcode EVCODE
(str) comma separated event codes for selection as OR combination; any negative code inverts selection, default = None
-i SEGIND, --segind SEGIND
(int) segment index for multi-panel detectors with raw.shape.ndim>2, default = 0
-S SLICE, --slice SLICE
(str) FOR DEBUGGING ONLY (str) slice of the panel image 2-k array selected for plots and pixel status, ex. "0:144,0:192", default = None
-r NRECS, --nrecs NRECS
(int) number of records to collect data, default = 1000
-w SHWIND, --shwind SHWIND
(str) window shape for feature 6 fitting to plane, ex. "15,15", default = 15,15
-R SNRMAX, --snrmax SNRMAX
(int) width of the good region in terms on number of spread/rms, default = 8.0
-t CTYPE, --ctype CTYPE
(str) type of calibration constants to save, default = status_data
-L LOGMODE, --logmode LOGMODE
(str) logging mode, one of CRITICAL, FATAL, ERROR, WARN, WARNING, INFO, DEBUG, NOTSET, default = INFO
-o DIRREPO, --dirrepo DIRREPO
(str) repository for calibration results, default = /sdf/group/lcls/ds/ana/detector/calib/constants/
-F FEATURES, --features FEATURES
(str) comma-separated list of Features [JAC-2022, Sadri, Automatic bad pixel mask maker...] from 1 to 6 to evaluate bad pixels, default = 1,2,3
--dirmode DIRMODE (int) mode for all mkdir, default = 0o2775
--filemode FILEMODE (int) mode for all saved files, default = 0o664
--group GROUP (str) group ownership for all files, default = ps-users
--databits DATABITS (int) data bits in ADC for code of intensity, default = 0o37777
--gainbits GAINBITS (int) gain mode switch bits in ADC, default = 0o0
--gmode GMODE (str) gain mode name-suffix for multi-gain detectors with raw.shape.ndim>3, ex: AHL-H, default = None
Dark processing
dark data processing for Feat.1 & 2 & 3, Ref.1
det_pixel_status -k exp=xpplw3319:run=287 -d XppGon.0:Epix100a.3 -F 1,2,3 -n 1000
These algorithms almost equivalent to pixel_status evaluarion in psana regular dark processing, command calibrun. It has different approach to definition of the good value range limits.
It works on collected block of frames shaped as (<number-of-frames>, <2-d-frame-size>) which implies limits on consumed memory.
[I] UtilsPixelStatus.py L0389
Summary of the bad pixel status evaluation for SNR=8.00, status_dark array
0o1: 0 / 540672 ( 0.000%) pixels Feat.2 mean <= 2933.437
0o2: 33 / 540672 ( 0.006%) pixels Feat.2 mean >= 5029.965
0o4: 0 / 540672 ( 0.000%) pixels Feat.3 std <= 1.525
0o10: 5314 / 540672 ( 0.983%) pixels Feat.3 std >= 4.053
Any bad status bit: 5319 / 540672 ( 0.984%) pixels
[I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_dark/epix100a_0009_20210716061106_xpplw3319_r0287_status_dark.dat
[I] UtilsPixelStatus.py L0639 Consumed time 19.674 sec
[I] RepoManager.py L0255
move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T115328_log_det_pixel_status_dubrovin.txt
to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T115328_log_det_pixel_status_dubrovin.txt
and create link
Light processing with fitting light background shape
light data processing for Feat.1 & 6, Ref.1
det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 1,6 -n 1000
It works on collected block of frames shaped as (<number-of-frames>, <2-d-frame-size>) which implies limits on consumed memory.
[I] UtilsPixelStatus.py L0389
Summary of the bad pixel status evaluation for SNR=8.00, status_light array
0o20: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -881.291
0o40: 33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 882.400
0o100: 0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.954
0o200: 4925 / 540672 ( 0.911%) pixels Feat.6 res_spr >= 2.613
Any bad status bit: 4930 / 540672 ( 0.912%) pixels
[I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_light/epix100a_0009_20210716073950_xpplw3319_r0293_status_light.dat
[I] UtilsPixelStatus.py L0639 Consumed time 3443.990 sec
[I] RepoManager.py L0255
move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T101656_log_det_pixel_status_dubrovin.txt
to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T101656_log_det_pixel_status_dubrovin.txt
and create link
light data processing for Feat.11, Ref.1
det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 11, -n 1000000
This algorithm accumulates array of max values over large number of events and analyzes distribution of max values over the frame. Should be processed separately from all other features on entire/large set of events, -n 1000000.
[I] UtilsPixelStatus.py L0389
Summary of the bad pixel status evaluation for SNR=8.00, status_max array
0o400: 7737 / 540672 ( 1.431%) pixels Feat.11 max-peds <= 114.447
0o1000: 3106 / 540672 ( 0.574%) pixels Feat.11 max-peds >= 216.111
Any bad status bit: 10843 / 540672 ( 2.005%) pixels
[I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424
[I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_max/epix100a_0009_20210716073950_xpplw3319_r0293_status_max.dat
[I] UtilsPixelStatus.py L0639 Consumed time 113.834 sec
[I] RepoManager.py L0255
move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T120040_log_det_pixel_status_dubrovin.txt
to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T120040_log_det_pixel_status_dubrovin.txt
and create link
Deployment of status_extra constants
For detail see Ref.6 Detector Calibration Constants Deployment, e.g.:
deploy_constants -x exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -C status_extra -D # -c ./calib -R 1-end
merges all status_* constants to status_merged in repository and deploys under /sdf/data/lcls/ds/xpp/xpplw3319/calib/Epix100a::CalibV1/XppGon.0:Epix100a.3/status_extra/
Release
Code available since ana-4.0.53-py3
# on sdf psana node:
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate ana-4.0.53-py3
References
- Bad pixel mask for epix100a xpplw3319
- Paper in JAC: Automatic bad-pixel mask maker for X-ray pixel detectors with application to serial crystallography,
- Journal of Applied Crystallography - 2022 - Sadri - Automatic bad‐pixel mask maker for X‐ray pixel detectors with.pdf
- Bad Pixel Status
- Calibration Scripts Repository and Logging
- Detector Calibration Constants Deployment