det_pixel_status
processes dark or light data and produces 2-d image or per-panel pixel status files.All algorithms/features are wrapped under the common hood of event processing loop and files' I/O. But, not all algorithms/features can be applied simultaneously due to different data types (dark or light) and different processing algorithms. Suggested combinations for -F parameters are shown in three examples below features with associated type of generated status constants in repository
Location of repository and calibration files can be found in the log file, see also Ref.5.
ana-4.0.51-py3 [username@sdfiana001:~/]$ det_pixel_status -h usage: det_pixel_status works with 2-d raw arrays ONLY! For 3-d the segment index --segind needs to be specified det_pixel_status -k <dataset> -d <detname> ... Ex.1: det_pixel_status -k exp=xpplw3319:run=287 -d XppGon.0:Epix100a.3 -F 1,2,3 -n 1000 # dark processing for Feat.1 & 2 & 3 Ex.2: det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 1,6 -n 1000 # light processing for Feat.1 & 6 Ex.3: det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 11, -n 1000000 # light processing for Feat.11 Feature # stands for 1: mean intensity of frames in good range 2: dark mean in good range 3: dark rms in good range 4: NOT-IMPLEMENTED for gain stage indicator values 5: NOT-IMPLEMENTED for gain stage indicator values 6: light average SNR of pixels over time 11: light intensity max-peds in good range - should be processed separately from all other features on entire/large set of events. Help: det_pixel_status -h Evaluates pixel status of any detector/segment using raw dark/light data optional arguments: -h, --help show this help message and exit -k DSKWARGS, --dskwargs DSKWARGS (str) DataSource parameters, default = exp=xpplw3319:run=293 -d DETNAME, --detname DETNAME (str) detector/detname name, default = epix_alc3 -n EVENTS, --events EVENTS (int) maximal number of events total (in runs, steps), default = 1000 -m EVSKIP, --evskip EVSKIP (int) number of events to skip in the beginning of each step, default = 0 -N STEPS, --steps STEPS (int) maximal number of steps or all by default, default = None -M STSKIP, --stskip STSKIP (int) number of steps to skip before processing or process all by default, default = None -c EVCODE, --evcode EVCODE (str) comma separated event codes for selection as OR combination; any negative code inverts selection, default = None -i SEGIND, --segind SEGIND (int) segment index for multi-panel detectors with raw.shape.ndim>2, default = 0 -S SLICE, --slice SLICE (str) FOR DEBUGGING ONLY (str) slice of the panel image 2-k array selected for plots and pixel status, ex. "0:144,0:192", default = None -r NRECS, --nrecs NRECS (int) number of records to collect data, default = 1000 -w SHWIND, --shwind SHWIND (str) window shape for feature 6 fitting to plane, ex. "15,15", default = 15,15 -R SNRMAX, --snrmax SNRMAX (int) width of the good region in terms on number of spread/rms, default = 8.0 -t CTYPE, --ctype CTYPE (str) type of calibration constants to save, default = status_data -L LOGMODE, --logmode LOGMODE (str) logging mode, one of CRITICAL, FATAL, ERROR, WARN, WARNING, INFO, DEBUG, NOTSET, default = INFO -o DIRREPO, --dirrepo DIRREPO (str) repository for calibration results, default = /sdf/group/lcls/ds/ana/detector/calib/constants/ -F FEATURES, --features FEATURES (str) comma-separated list of Features [JAC-2022, Sadri, Automatic bad pixel mask maker...] from 1 to 6 to evaluate bad pixels, default = 1,2,3 --dirmode DIRMODE (int) mode for all mkdir, default = 0o2775 --filemode FILEMODE (int) mode for all saved files, default = 0o664 --group GROUP (str) group ownership for all files, default = ps-users --databits DATABITS (int) data bits in ADC for code of intensity, default = 0o37777 --gainbits GAINBITS (int) gain mode switch bits in ADC, default = 0o0 --gmode GMODE (str) gain mode name-suffix for multi-gain detectors with raw.shape.ndim>3, ex: AHL-H, default = None |
dark data processing for Feat.1 & 2 & 3, Ref.1
det_pixel_status -k exp=xpplw3319:run=287 -d XppGon.0:Epix100a.3 -F 1,2,3 -n 1000
These algorithms almost equivalent to pixel_status evaluarion in psana regular dark processing, command calibrun
. It has different approach to definition of the good value range limits.
It works on collected block of frames shaped as (<number-of-frames>, <2-d-frame-size>) which implies limits on consumed memory.
[I] UtilsPixelStatus.py L0389 Summary of the bad pixel status evaluation for SNR=8.00, status_dark array 0o1: 0 / 540672 ( 0.000%) pixels Feat.2 mean <= 2933.437 0o2: 33 / 540672 ( 0.006%) pixels Feat.2 mean >= 5029.965 0o4: 0 / 540672 ( 0.000%) pixels Feat.3 std <= 1.525 0o10: 5314 / 540672 ( 0.983%) pixels Feat.3 std >= 4.053 Any bad status bit: 5319 / 540672 ( 0.984%) pixels [I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424 [I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_dark/epix100a_0009_20210716061106_xpplw3319_r0287_status_dark.dat [I] UtilsPixelStatus.py L0639 Consumed time 19.674 sec [I] RepoManager.py L0255 move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T115328_log_det_pixel_status_dubrovin.txt to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T115328_log_det_pixel_status_dubrovin.txt and create link |
light data processing for Feat.1 & 6, Ref.1
det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 1,6 -n 1000
It works on collected block of frames shaped as (<number-of-frames>, <2-d-frame-size>) which implies limits on consumed memory.
[I] UtilsPixelStatus.py L0389 Summary of the bad pixel status evaluation for SNR=8.00, status_light array 0o20: 0 / 540672 ( 0.000%) pixels Feat.6 res_med <= -881.291 0o40: 33 / 540672 ( 0.006%) pixels Feat.6 res_med >= 882.400 0o100: 0 / 540672 ( 0.000%) pixels Feat.6 res_spr <= 0.954 0o200: 4925 / 540672 ( 0.911%) pixels Feat.6 res_spr >= 2.613 Any bad status bit: 4930 / 540672 ( 0.912%) pixels [I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424 [I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_light/epix100a_0009_20210716073950_xpplw3319_r0293_status_light.dat [I] UtilsPixelStatus.py L0639 Consumed time 3443.990 sec [I] RepoManager.py L0255 move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T101656_log_det_pixel_status_dubrovin.txt to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T101656_log_det_pixel_status_dubrovin.txt and create link |
light data processing for Feat.11, Ref.1
det_pixel_status -k exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -F 11, -n 1000000
This algorithm accumulates array of max values over large number of events and analyzes distribution of max values over the frame. Should be processed separately from all other features on entire/large set of events, -n 1000000
.
[I] UtilsPixelStatus.py L0389 Summary of the bad pixel status evaluation for SNR=8.00, status_max array 0o400: 7737 / 540672 ( 1.431%) pixels Feat.11 max-peds <= 114.447 0o1000: 3106 / 540672 ( 0.574%) pixels Feat.11 max-peds >= 216.111 Any bad status bit: 10843 / 540672 ( 2.005%) pixels [I] RepoManager.py L0176 panel directory: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424 [I] UtilsCalib.py L0150 saved: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_max/epix100a_0009_20210716073950_xpplw3319_r0293_status_max.dat [I] UtilsPixelStatus.py L0639 Consumed time 113.834 sec [I] RepoManager.py L0255 move logfile: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/2023/2023-10-09T120040_log_det_pixel_status_dubrovin.txt to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/2023/2023-10-09T120040_log_det_pixel_status_dubrovin.txt and create link |
For detail see Ref.6 Detector Calibration Constants Deployment, e.g.:
deploy_constants -x exp=xpplw3319:run=293 -d XppGon.0:Epix100a.3 -C status_extra -D # -c ./calib -R 1-end
merges all status_* constants to status_merged in repository and deploys under /sdf/data/lcls/ds/xpp/xpplw3319/calib/Epix100a::CalibV1/XppGon.0:Epix100a.3/status_extra/
Code available since ana-4.0.53-py3
# on sdf psana node: source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh conda deactivate conda activate ana-4.0.53-py3 |