Page History
...
Evolution of calibration system of LCLS creates new opportunities and increase complexity. Large number of detectors and scripts for calibration processing makes it hard to track down what is going on. This note is intended to collect in one spot references to location of all intermediate repositories and log files.
Environment variables
psana/psdm
lcls1
source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh, setup_testrel
export SIT_ROOT=/reg/g/psdm/
export SIT_DATA=/cds/group/psdm/data/
export SIT_PSDM_DATA=/cds/data/psdm/
in Detector/src/dir_root.py
DIR_ROOT = os.getenv('SIT_ROOT') # /reg/g/psdm/
DIR_LOG_AT_START = /cds/group/psdm/detector/logs/atstart/
DIR_REPO = /cds/group/psdm/detector/calib/constants/
lclc2
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
export DIR_PSDM=/cds/group/psdm # or /sdf/group/psdm # s3df
in lcls2/psana/psana/detector/dir_root.py
DIR_ROOT = os.getenv('DIR_PSDM')
DIR_LOG_AT_START = /cds/group/psdm/detector/logs/atstart/
DIR_REPO = /cds/group/psdm/detector/calib2/constants/
s3df
lcls1
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh, setup_testrel
export SIT_ROOT=/sdf/group/lcls/ds/ana/
export SIT_DATA=/sdf/group/lcls/ds/ana/data/
export SIT_PSDM_DATA=/sdf/data/lcls/ds/
lclc2
source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh
Repositories
Structure
- Log files at start, in the script level directory, and detector-type level directory:
<DIR_LOG_AT_START>/<year>/<year>_lcls<1-or-2>_<script-name>.txt
<DIR_REPO>/scripts/<script-name>/logs/<year>/<time-stamp-of-processing>_log_<script-name>_<uid>.txt@ <<< this is a reference if processing script is completed.
<DIR_REPO>/<dettype>/logs/<year>/<time-stamp-of-processing>_log_<script-name>_<uid>.txt
- panel aliases and per-panel calibration files:
<DIR_REPO>/<dettype>/.aliases.txt
<DIR_REPO>/<dettype>/<panel-id>/<calib-type>/<dettype>_<det-id-alias>_<YYYYMMSS<6-digit-μs>>_<experiment-name>_r<run-number>_<calib-type>.dat
Examples
/cds/group/psdm/detector/logs/atstart/2023/2023_lcls1_det_raw_pixel_status.txt
/cds/group/psdm/detector/calib/constants/scripts/det_raw_pixel_status/logs/2023/2023-04-21T103547_log_det_raw_pixel_status_dubrovin.txt@ <<< reference to:
/cds/group/psdm/detector/calib/constants/epix100a/logs/2023/2023-04-21T103547_log_det_raw_pixel_status_dubrovin.txt
Code Block | ||||
---|---|---|---|---|
| ||||
0001 3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424 2023-04-21T08:26:51 xpplw3319 r0287 XppGon.0:Epix100a.3 dubrovin
0002 3925999620-0996579585-0553648138-1232098304-1221641739-2650251521-3976200215 2023-04-21T11:10:23 xpplw3319 r0287 XppGon.0:Epix100a.2 dubrovin
0003 3925999620-0996432897-3590324234-1232100352-1154532875-2654088449-0033554455 2023-04-21T11:10:33 xpplw3319 r0287 XppGon.0:Epix100a.1 dubrovin
0004 3925999620-1000248321-4076863498-1794137088-0410306571-2402454273-0067108888 2023-04-21T11:10:46 xpplw3319 r0287 XppGon.0:Epix100a.4 dubrovin |
/cds/group/psdm/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_data/epix100a_0001_20210716061106_xpplw3319_r0287_status_data.dat
LCLS scripts
script | record at start | repository | logs | control parameters | comments | |||
---|---|---|---|---|---|---|---|---|
Generic area detectors | ||||||||
calibrun | >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibrun.txt | >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/calibrun/ | /reg/g/psdm/logs/calibman/<year>/<month>/<repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp> -_log -<login>-<job-id>_calibrun_<login>.txt <repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt | N/A | CLI dark processing for generic detector | |||
calibfile | /reg/g/psdm/logs/calibman/<year>/<month>/calibfile.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibfile.txt | >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/calibfile/ | >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp>_log_calibfile_<login>.txt <repository>/logs/2022/2022-10- log record on copy file13T170608_log_calibfile_dubrovin.txt | -l, --lfname | CLI for calib file deployment under .../<experiment>/calib/... | DEPRECATED calibman_deprecated | ||
geometry_deploy_constants | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_geometry_deploy_constants.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_geometry_deploy_constants.txt ./work | /reg/g/psdm/ logs/calibman/<year>/<month>/<time-stamp>-log-<login>-<job-id>.txt | N/A | GUI for generic detector - dark processing, file manager, image browser etc. Why it is deprecated? Beside dark processing it has many other useful maintenance and browsing options. | ||||
epix10ka | ||||||||
detector/calib/geometry/ >ana-4.0.46-py3: $SIT_ROOT/detector/calib/geometry | <repository>/<dettype>/<dettype>_<full-lcls1-detector-name>_<time-stamp>.data >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp>_log_geometry_deploy_constants_<login>.txt <repository>/logs/2022/2022-10-14T094805_log_geometry_deploy_constants_dubrovin.txt | -o, --dirrepo | Geometry calibration constants deployment CLI | |||||
dcs | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_dcs.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/<year>/<year>_lcls1_dcs.txt | /reg/g/psdm/<INS>/<experiment>/calib/<dettype> # default experiment-detector repo or parameter -c <path to local calib> e.g.: -c /reg/d/psdm/detector/calib # detector-dependent repo >ana-4.0.46-py3: $SIT_PSDM_DATA/XPP/xpptut15/calib/epix100a | run-time only | Interaction with dcs-store - hdf5 file calibration constants. | ||||
deploy_constants for releases > ana-4.0.42 | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_deploy_constants.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_deploy_constants.txt | /reg/g/psdm/detector/calib/constants/ >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/ | <repository>/logs/<year>/<time-stamp>_log_deploy_constants_<uid>.txt <repository>/logs/2022/2022-10-14T105634_log_deploy_constants_dubrovin.txt | -o , --dirrepo | Deployment of default constants for all generic (one gain range) detectors. Special case of deployment with merging at -t status_extra | |||
det_pixel_status | $SIT_ROOT/detector/logs/atstart/<year>/<year>_lcls1_det_pixel_status.txt /sdf/group/lcls/ds/ana/detector/logs/atstart/2023/2023_lcls1_det_pixel_status.txt | $SIT_ROOT/detector/calib/constants/ | begins with: /sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/<year>/<time-stamp>_log_det_pixel_status_<uid>.txt | -o, --dirrepo | Light data processing for bad pixel status. See: Bad pixel mask ... | |||
epix10ka | ||||||||
epix10ka_offset_calibration | <repository>/ | epix10ka_offset_calibration | /reg/g/psdm/detector/gains/epix10k/panels/ | in repository logs/<year>_epix10ka_offset_calibration.txt - log record at start >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt | /reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_offset_calibration_<login>.txt ???? | -o, --dirrepo | Processing of the charge injection runs with 103 steps/calib-cycles |
epix10ka_pedestals_calibration | <repository>/ | reg/g/psdm/detector/gains/epix10k/panels/logs/<year>_epix10ka_pedestals_calibration.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt | /reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/ | logs/<year>/<time-stamp>_log_epix10ka_pedestals_calibration_<login>.txt | -o, --dirrepo | Dark processing of runs with 5 steps/calib-cycles | |
epix10ka_deploy_constants | <repository>/ | reg/g/psdm/detector/gains/epix10k/panels/logs/<year>_epix10ka_deploy_constants.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_deploy_constants.txt | - log record at start/reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_deploy_constants_<login>.txt | -o, --dirrepo | Merging and deployment ofper-panel constants. | ||
Jungfrau | ||||||||
jungfrau_dark_proc | <repository>/logs/<year>_log_jungfrau_dark_proc.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_dark_proc.txt | /reg/g/psdm/detector/gains/jungfrau/panels/ contains per-panel constants | in repository logs/<year>_log_jungfrau_dark_proc.txt - log record at start <repository>/logs/<year>/<time-stamp>_jungfrau_dark_proc_<login>.txt | -o, --dirrepo | replacement for jungfrau_ndarr_dark_proc intended for per-panel calibration constants. Dark processing of runs with 3 steps/calib-cycles | |||
jungfrau_deploy_constants | <repository>/ reg/g/psdm/detector/gains/jungfrau/panels/logs/<year>_log_jungfrau_deploy_constants.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1 in repository logs/<year>_log_jungfrau_deploy_constants.txt - log record at start | /reg/g/psdm/detector/gains/jungfrau/panels/ | <repository>/logs/<year>/<time-stamp>_jungfrau_deploy_constants_<login>.txt | -o, --dirrepo | works after jungfrau_dark_proc to merge and deploy constants under .../<experiment>/calib/... | |||
jungfrau_gain_constants | >ana-4.0.44 /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_gain_constants.txt | /reg/g/psdm/detector/gains/jungfrau/ - reference to dirs with gain constants for panel-ids | /reg/g/psdm/detectorlogs/calibcalibman/jungfrau_gain/jungfrau-<panel-id>.h5<year>/<time-stamp>-<experiment>-<run>-<login>.txt for >ana-4.0.44: /reg/g/psdm/ logsdetector/ calibmangains/jungfrau _gain/logs/<year>/<time-stamp> -_log_jungfrau_gain_constants_<experiment>- <run>-r<run>_<login>.txt | -R, --repdir | ||||
DEPRECATED jungfrau_ndarr_dark_proc | /reg/g/psdm/logs/calibman/<year>/<month>/jungfrau_ndarr_dark_proc.txt - log record at start | Dark processing and deployment of calibration constants for etire detector. It was DEPRECATED due to new per-panel approach to calibration constants. Replaced by jungfrau_dark_proc and jungfrau_deploy_constants |
LCLS2 scripts
script | record at start | repository | logs | control parameters | comments | |||||
---|---|---|---|---|---|---|---|---|---|---|
Generic calibration manager | ||||||||||
calibman | /cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_calibman.txt | /cds/group/psdm/detector/calib2/constants/ DEPRECATED: /lcls2/cds/group/psdm/logs/calibman/lcls2 | <repository>/{scripts/<script-name> OR <dettype>}/logs/<year>/<time-stamp>-<script-name>-<login>.txt | -L, --logdir | Interaction with MongoDB etc | |||||
epix10ka | ||||||||||
epix10ka_pedestals_calibration | /cds/group/psdm/ | detectorlogs/ | gains2atstart/ | epix10ka/<year>/<year>_logrec_epix10ka_pedestals_calibration | _<year>.txt - log record at start.txt | /cds/group/psdm/detector/calib2/constants/ DEPRECATED: /cds/group/psdm/detector/gains2/epix10ka/panels/ | <repository>/{scripts/<script-name> OR <dettype>}/ | logs/<year>/<time-stamp>-<script-name>-<login>.txt | -o, --dirrepo | |
epix10ka_deploy_constants | /cds/group/psdm/ detectorlogs/ gains2atstart/ epix10ka/in repository logs/log<year>/<year>_logrec_epix10ka_deploy_constants_<year>.txt - log record at start TODO: move year ahead, add saving entire log under .txt | /cds/group/psdm/detector/calib2/constants/ DEPRECATED: /cds/group/psdm/detector/gains2/epix10ka/panels/ | <repository>/{scripts/<script-name> OR <dettype>}/logs/<year>/<time-stamp>-<script-name>-<login>.txt | -o, --dirrepo | ||||||
epix10ka_charge_injection | /cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_epix10ka_charge_injection.txt | /cds/group/psdm/detector/calib2/constants/ | ||||||||
epix100, opal | ||||||||||
det_dark_proc | /cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_det_dark_proc. /worktxt DEPRECATED: /cds/group/psdm/logs/atstart/<year>/<year>_log_det_dark_proc.txt | /cds/group/psdm/detector/calib2/constants/ | <repository>/{scripts/<script-name> OR <dettype>}/ in repository logs/<year>/<time-stamp>-<script-name>-<login>.txt ??? | |||||||
Tentative logging for LCLS2
...
/cds/group/psdm/logs/atstart/<year>/<year>-loglogrec-on-start-<script-name>.txt - single file per year per script.
...
where <detector-type> e.g. epix10ka
...
Logging files
/cds/group/psdm/detector/gains2/<detector-type>/panels/logs/<year>/<time-stamp>-<script-name>-<login>.txt
...