Intro
Evolution of calibration system of LCLS creates new opportunities and increase complexity. Large number of detectors and scripts for calibration processing makes it hard to track down what is going on. This note is intended to collect in one spot references to location of all intermediate repositories and log files.
Environment variables
psana/psdm
lcls1
source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh, setup_testrel
export SIT_ROOT=/reg/g/psdm/
export SIT_DATA=/cds/group/psdm/data/
export SIT_PSDM_DATA=/cds/data/psdm/
lclc2
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
export DIR_PSDM=/cds/group/psdm
DIR_ROOT = os.getenv('DIR_PSDM')
DIR_LOG_AT_START = /cds/group/psdm/detector/logs/atstart/
DIR_REPO = /cds/group/psdm/detector/calib2/constants/
s3df
lcls1
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh, setup_testrel
export SIT_ROOT=/sdf/group/lcls/ds/ana/
export SIT_DATA=/sdf/group/lcls/ds/ana/data/
export SIT_PSDM_DATA=/sdf/data/lcls/ds/
lclc2
source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.shLCLS scripts
| script | record at start | repository | logs | control parameters | comments |
|---|---|---|---|---|---|
| area detectors | |||||
calibrun | >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibrun.txt | >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/calibrun/ | /reg/g/psdm/logs/calibman/<year>/<month>/<repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp>_log_calibrun_<login>.txt <repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt | N/A | CLI dark processing for generic detector |
| calibfile | /reg/g/psdm/logs/calibman/<year>/<month>/calibfile.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibfile.txt | >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/calibfile/ | >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp>_log_calibfile_<login>.txt <repository>/logs/2022/2022-10-13T170608_log_calibfile_dubrovin.txt | -l, --lfname | CLI for calib file deployment under .../<experiment>/calib/... |
| geometry_deploy_constants | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_geometry_deploy_constants.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_geometry_deploy_constants.txt | /reg/g/psdm/detector/calib/geometry/ >ana-4.0.46-py3: $SIT_ROOT/detector/calib/geometry | <repository>/<dettype>/<dettype>_<full-lcls1-detector-name>_<time-stamp>.data >ana-4.0.46-py3: <repository>/logs/<year>/<time-stamp>_log_geometry_deploy_constants_<login>.txt <repository>/logs/2022/2022-10-14T094805_log_geometry_deploy_constants_dubrovin.txt | -o, --dirrepo | Geometry calibration constants deployment CLI |
| dcs | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_dcs.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/<year>/<year>_lcls1_dcs.txt | /reg/g/psdm/<INS>/<experiment>/calib/<dettype> # default experiment-detector repo or parameter -c <path to local calib> e.g.: -c /reg/d/psdm/detector/calib # detector-dependent repo >ana-4.0.46-py3: $SIT_PSDM_DATA/XPP/xpptut15/calib/epix100a | run-time only | Interaction with dcs-store - hdf5 file calibration constants. | |
deploy_constants for releases > ana-4.0.42 | /reg/g/psdm/logs/atstart/<year>/<year>_lcls1_deploy_constants.txt >ana-4.0.46-py3: $SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_deploy_constants.txt | /reg/g/psdm/detector/calib/constants/ >ana-4.0.46-py3: $SIT_ROOT/detector/calib/constants/ | <repository>/logs/<year>/<time-stamp>_log_deploy_constants_<uid>.txt <repository>/logs/2022/2022-10-14T105634_log_deploy_constants_dubrovin.txt | -o , --dirrepo | Deployment of default constants for all generic (one gain range) detectors. Special case of deployment with merging at -t status_extra |
| det_raw_pixel_status | /reg/g/psdm/detector/logs/atstart/2023/2023_lcls1_det_raw_pixel_status.txt | $SIT_ROOT/detector/calib/constants/ | /reg/g/psdm/detector/calib/constants/logs/<year>/<time-stamp>_log_det_raw_pixel_status_<uid>.txt /reg/g/psdm/detector/calib/constants/epix100a/logs/<year>/<time-stamp>_log_det_raw_pixel_status_<uid>.txt | -o, --dirrepo | Light data processing for bad pixel status. See: Bad pixel mask ... |
| epix10ka | |||||
| epix10ka_offset_calibration | <repository>/logs/<year>_epix10ka_offset_calibration.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt | /reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_offset_calibration_<login>.txt ???? | -o, --dirrepo | Processing of the charge injection runs with 103 steps/calib-cycles |
| epix10ka_pedestals_calibration | <repository>/logs/<year>_epix10ka_pedestals_calibration.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt | /reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_pedestals_calibration_<login>.txt | -o, --dirrepo | Dark processing of runs with 5 steps/calib-cycles |
| epix10ka_deploy_constants | <repository>/logs/<year>_epix10ka_deploy_constants.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_deploy_constants.txt | /reg/g/psdm/detector/gains/epix10k/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_deploy_constants_<login>.txt | -o, --dirrepo | Merging and deployment ofper-panel constants. |
| Jungfrau | |||||
| jungfrau_dark_proc | <repository>/logs/<year>_log_jungfrau_dark_proc.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_dark_proc.txt | /reg/g/psdm/detector/gains/jungfrau/panels/ contains per-panel constants | <repository>/logs/<year>/<time-stamp>_jungfrau_dark_proc_<login>.txt | -o, --dirrepo | replacement for jungfrau_ndarr_dark_proc intended for per-panel calibration constants. Dark processing of runs with 3 steps/calib-cycles |
| jungfrau_deploy_constants | <repository>/logs/<year>_log_jungfrau_deploy_constants.txt >ana-4.0.37: /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_deploy_constants.txt | /reg/g/psdm/detector/gains/jungfrau/panels/ | <repository>/logs/<year>/<time-stamp>_jungfrau_deploy_constants_<login>.txt | -o, --dirrepo | works after jungfrau_dark_proc to merge and deploy constants under .../<experiment>/calib/... |
| jungfrau_gain_constants | >ana-4.0.44 /cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_gain_constants.txt | /reg/g/psdm/detector/gains/jungfrau/ - reference to dirs with gain constants for panel-ids | /reg/g/psdm/logs/calibman/jungfrau_gain/<year>/<time-stamp>-<experiment>-<run>-<login>.txt for >ana-4.0.44: /reg/g/psdm/detector/gains/jungfrau/logs/<year>/<time-stamp>_log_jungfrau_gain_constants_<experiment>-r<run>_<login>.txt | -R, --repdir | |
DEPRECATED jungfrau_ndarr_dark_proc | /reg/g/psdm/logs/calibman/<year>/<month>/jungfrau_ndarr_dark_proc.txt - log record at start | Dark processing and deployment of calibration constants for etire detector. It was DEPRECATED due to new per-panel approach to calibration constants. Replaced by jungfrau_dark_proc and jungfrau_deploy_constants | |||
LCLS2 scripts
| script | record at start | repository | logs | control parameters | comments |
|---|---|---|---|---|---|
| Generic calibration manager | |||||
| calibman | /cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_calibman.txt DEPRECATED: /cds/group/psdm/logs/atstart/2022/2022_logrec_calibman.txt | /cds/group/psdm/detector/calib2/constants/ DEPRECATED: /cds/group/psdm/logs/calibman/lcls2 | <repository>/<script-or-dettype>/logs/<year>/<time-stamp>-<script-name>-<login>.txt e.g. /cds/group/psdm/logs/calibman/lcls2/2022/20220202T081010-calibman-dubrovin.txt | -L, --logdir | Interaction with MongoDB etc |
| epix10ka | |||||
| epix10ka_pedestals_calibration | /cds/group/psdm/logs/atstart/<year>/<year>_logrec_epix10ka_pedestals_calibration.txt DEPRECATED: <repository>/panels/logs/log_pedestals_calibration_<year>.txt | /cds/group/psdm/detector/gains2/epix10ka/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_pedestals_calibration_<login>.txt e.g. /cds/group/psdm/detector/gains2/epix10ka/panels/logs/2022/2022-02-02T115201_log_epix10ka_pedestals_calibration_dubrovin.txt | -o, --dirrepo | |
| epix10ka_deploy_constants | /cds/group/psdm/logs/atstart/<year>/<year>_logrec_epix10ka_deploy_constants.txt e.g. /cds/group/psdm/logs/atstart/2022/2022_logrec_epix10ka_deploy_constants.txt DEPRECATED: <repository>/panels/logs/log_deploy_constants_<year>.txt | /cds/group/psdm/detector/gains2/epix10ka/panels/ | <repository>/logs/<year>/<time-stamp>_log_epix10ka_deploy_constants_<login>.txt e.g. /cds/group/psdm/detector/gains2/epix10ka/panels/logs/2022/2022-02-02T120758_log_epix10ka_deploy_constants_dubrovin.txt | -o, --dirrepo | |
| epix100, opal | |||||
det_dark_proc | /cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_det_dark_proc.txt DEPRECATED: /cds/group/psdm/logs/atstart/<year>/<year>_log_det_dark_proc.txt | /cds/group/psdm/detector/calib2/constants/ | <repository>/<script-or-dettype>/logs/<year>/<time-stamp>_log_det_dark_proc_<login>.txt e.g. /cds/group/psdm/detector/calib2/constants/epix100/logs/2022/2022-02-02T080749_log_det_dark_proc_dubrovin.txt | ||
Tentative logging for LCLS2
Q: is it ok to save logs in repo?
A: if repodir is changed path will be loss along with local location. It would be nice to have record at start in well known static directory. Each record points to the main log file etc.
Logging at start records
Is intended to keep all records at start for all scripts in a single place.
Root directory is intended to be hardwired, e.g.
/cds/group/psdm/logs/atstart/
Further path
/cds/group/psdm/logs/atstart/<year>/<year>-logrec-<script-name>.txt - single file per year per script.
Repository
Root directory
-o, --dirrepo
/cds/group/psdm/detector/gains2/<detector-type>/panels/
/cds/group/psdm/detector/gains2/<detector-type>/panels/<panel-id>,
where <detector-type> e.g. epix10ka
Logging files
/cds/group/psdm/detector/gains2/<detector-type>/panels/logs/<year>/<time-stamp>-<script-name>-<login>.txt
References
Overview
Content Tools