Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Evolution of calibration system of LCLS creates new opportunities and increase complexity. Large number of detectors and scripts for calibration processing makes it hard to track down what is going on. This note is intended to collect in one spot references to location of all intermediate repositories and log files.

Environment variables

psana/psdm

lcls1

source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh, setup_testrel

export SIT_ROOT=/reg/g/psdm/
export SIT_DATA=/cds/group/psdm/data/
export SIT_PSDM_DATA=/cds/data/psdm/

in Detector/src/dir_root.py

DIR_ROOT = os.getenv('SIT_ROOT') # /reg/g/psdm/

DIR_LOG_AT_START = /cds/group/psdm/detector/logs/atstart/

DIR_REPO = /cds/group/psdm/detector/calib/constants/

lclc2

source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh

export DIR_PSDM=/cds/group/psdm # or /sdf/group/psdm  # s3df

in lcls2/psana/psana/detector/dir_root.py

DIR_ROOT = os.getenv('DIR_PSDM')

DIR_LOG_AT_START = /cds/group/psdm/detector/logs/atstart/

DIR_REPO = /cds/group/psdm/detector/calib2/constants/

s3df

lcls1

source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh, setup_testrel

export SIT_ROOT=/sdf/group/lcls/ds/ana/
export SIT_DATA=/sdf/group/lcls/ds/ana/data/
export SIT_PSDM_DATA=/sdf/data/lcls/ds/

lclc2

source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh

Repositories

Structure

  • Log files at start, in the script level directory, and detector-type level directory:

<DIR_LOG_AT_START>/<year>/<year>_lcls<1-or-2>_<script-name>.txt

<DIR_REPO>/scripts/<script-name>/logs/<year>/<time-stamp-of-processing>_log_<script-name>_<uid>.txt@ <<< this is a reference if processing script is completed.

<DIR_REPO>/<dettype>/logs/<year>/<time-stamp-of-processing>_log_<script-name>_<uid>.txt

  • panel aliases and per-panel calibration files:

<DIR_REPO>/<dettype>/.aliases.txt

<DIR_REPO>/<dettype>/<panel-id>/<calib-type>/<dettype>_<det-id-alias>_<YYYYMMSS<6-digit-μs>>_<experiment-name>_r<run-number>_<calib-type>.dat

Examples

/cds/group/psdm/detector/logs/atstart/2023/2023_lcls1_det_raw_pixel_status.txt

/cds/group/psdm/detector/calib/constants/scripts/det_raw_pixel_status/logs/2023/2023-04-21T103547_log_det_raw_pixel_status_dubrovin.txt@ <<< reference to:

/cds/group/psdm/detector/calib/constants/epix100a/logs/2023/2023-04-21T103547_log_det_raw_pixel_status_dubrovin.txt

Code Block
title/cds/group/psdm/detector/calib/constants/epix100a/.aliases.txt
collapsetrue
0001 3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424 2023-04-21T08:26:51  xpplw3319 r0287 XppGon.0:Epix100a.3 dubrovin
0002 3925999620-0996579585-0553648138-1232098304-1221641739-2650251521-3976200215 2023-04-21T11:10:23  xpplw3319 r0287 XppGon.0:Epix100a.2 dubrovin
0003 3925999620-0996432897-3590324234-1232100352-1154532875-2654088449-0033554455 2023-04-21T11:10:33  xpplw3319 r0287 XppGon.0:Epix100a.1 dubrovin
0004 3925999620-1000248321-4076863498-1794137088-0410306571-2402454273-0067108888 2023-04-21T11:10:46  xpplw3319 r0287 XppGon.0:Epix100a.4 dubrovin

/cds/group/psdm/detector/calib/constants/epix100a/3925999620-0996513537-2080374794-1794135040-0940361739-2398406657-0419430424/status_data/epix100a_0001_20210716061106_xpplw3319_r0287_status_data.dat

LCLS scripts

in repository

logs/<Year>_epix10ka_pedestals_calibration.txt - log record on start

logs/<Year>reg/g/psdm/detector/gains/epix10k/panels/

in repository

logs/<Year> - log record on startlogs/<Year>
scriptrecord at startrepositorylogscontrol parameterscommentsGeneric area detectors
area detectors




calibrun

>ana-4.0.46-py3:

$SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibrun.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/calib/constants/calibrun/calibrun

/reg/g/psdm/logs/calibman

/<Year>/<Month>

/<year>/<month>/<repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt

>ana-4.0.46-py3:

<repository>/logs/<year>/<time-stamp>

-

_log

-<login>-<job-id>

_calibrun_<login>.txt

<repository>/logs/2022/2022-10-12T103606_log_calibrun_dubrovin.txt

N/ACLI dark processing for generic detector
calibfile

/reg/g/psdm/logs/calibman

/<Year>/<Month>/calibfile.txt - log record on copy file

/<year>/<month>/calibfile.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_calibfile.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/calib/constants/calibfile/

>ana-4.0.46-py3:

<repository>/logs/<year>/<time-stamp>_log_calibfile_<login>.txt

<repository>/logs/2022/2022-10-13T170608_log_calibfile_dubrovin.txt

-l, --lfnameCLI for calib file deployment under .../<experiment>/calib/...

DEPRECATED

calibman_deprecated

geometry_deploy_constants

/reg/g/psdm/logs/atstart/<year>/<year>_lcls1_geometry_deploy_constants.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_geometry_deploy_constants.txt

./work

/reg/g/psdm

/logs/calibman/<Year>/<Month>/<time-stamp>-log-<login>-<job-id>.txt
N/AGUI for generic detector - dark processing, file manager, image browser etc. Why it is deprecated? Beside dark processing it has many other useful maintenance and browsing options.
epix10ka

/detector/calib/geometry/

>ana-4.0.46-py3:

$SIT_ROOT/detector/calib/geometry

<repository>/<dettype>/<dettype>_<full-lcls1-detector-name>_<time-stamp>.data

>ana-4.0.46-py3:

<repository>/logs/<year>/<time-stamp>_log_geometry_deploy_constants_<login>.txt

<repository>/logs/2022/2022-10-14T094805_log_geometry_deploy_constants_dubrovin.txt

-o, --dirrepoGeometry calibration constants deployment CLI
dcs

/reg/g/psdm/logs/atstart/<year>/<year>_lcls1_dcs.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/logs/atstart/<year>/<year>_lcls1_dcs.txt

/reg/g/psdm/<INS>/<experiment>/calib/<dettype> # default experiment-detector repo

or parameter -c <path to local calib>

e.g.:

-c /reg/d/psdm/detector/calib # detector-dependent repo

>ana-4.0.46-py3:

$SIT_PSDM_DATA/XPP/xpptut15/calib/epix100a
$SIT_PSDM_DATA/detector/calib/epix100a

run-time only
Interaction with dcs-store - hdf5 file calibration constants.

deploy_constants

for releases > ana-4.0.42

/reg/g/psdm/logs/atstart/<year>/<year>_lcls1_deploy_constants.txt

>ana-4.0.46-py3:

$SIT_ROOT/detector/logs/atstart/2022/2022_lcls1_deploy_constants.txt

epix10ka_offset_calibration

/reg/g/psdm/detector/calib/constants/

>ana-4.0.46-py3:

$SIT_ROOT/detector

/gains/epix10k/panels/

/calib/constants/

<repository>/logs/<year>/<time-stamp>_log_deploy_constants_<uid>.txt

<repository>/logs/2022/2022-10-14T105634_log_deploy_constants_dubrovin.txt

-o , --dirrepo Deployment of default constants for all generic (one gain range) detectors. Special case of deployment with merging at -t status_extra
det_pixel_status

$SIT_ROOT/detector/logs/atstart/<year>/<year>_lcls1_det_pixel_status.txt

/sdf/group/lcls/ds/ana/detector/logs/atstart/2023/2023_lcls1_det_pixel_status.txt

$SIT_ROOT/detector/calib/constants/

begins with:

/sdf/group/lcls/ds/ana/detector/calib/constants/scripts/det_pixel_status/logs/<year>/<time-stamp>_log_det_pixel_status_<uid>.txt
move at the end to: /sdf/group/lcls/ds/ana/detector/calib/constants/epix100a/logs/<year>/<time-stamp>_log_det_pixel_status_<uid>.txt

-o, --dirrepo Light data processing for bad pixel status. See: Bad pixel mask ...






epix10ka
epix10ka_offset_calibration

<repository>/logs/<year>

in repository

logs/<Year>_epix10ka_offset_calibration.txt - log record on start

>ana-4.0.37:

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt

/reg/g/psdm/detector/gains/epix10k/panels/

<repository>/logs/<year>logs/<Year>/<time-stamp>_log_epix10ka_offset_calibration_<login>.txt ????

-o, --dirrepoProcessing of the charge injection runs with 103 steps/calib-cycles
epix10ka_pedestals_calibration

<repository>/logs/<year>_epix10ka_pedestals_calibration.txt

>ana-4.0.37:

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_offset_calibration.txt

/reg/g/psdm/detector/gains/epix10k/panels/

<repository>/logs/<year>/<time-stamp>_log_epix10ka_pedestals_calibration_<login>.txt

-o, --dirrepoDark processing of runs with 5 steps/calib-cycles
epix10ka_deploy_constants

<repository>/

logs/<year>_epix10ka_deploy_constants.txt

>ana-4.0.37:

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1_epix10ka_deploy_constants.txt

/reg/g/psdm/detector/gains/epix10k/panels/

<repository>/logs/<year>

/<time-stamp>_log_epix10ka_deploy_constants_<login>.txt

-o, --dirrepoMerging and deployment ofper-panel constants.






Jungfrau
jungfrau_dark_proc

<repository>/logs/<year>_log_jungfrau_dark_proc.txt

>ana-4.0.37:

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_dark_proc.txt

/reg/g/psdm/detector/gains/jungfrau/panels/

contains per-panel constants

in repository

logs/<Year>_log_jungfrau_dark_proc.txt - log record on start

logs/<Year><repository>/logs/<year>/<time-stamp>_jungfrau_dark_proc_<login>.txt

-o, --dirrepo

replacement for jungfrau_ndarr_dark_proc

intended for per-panel calibration constants.

Dark processing of runs with 3 steps/calib-cycles

jungfrau_deploy_constants

<repository>/

reg/g/psdm/detector/gains/jungfrau/panels/

logs/<year>_log_jungfrau_deploy_constants.txt

>ana-4.0.37:

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1

in repository

logs/<Year>_log_jungfrau_deploy_constants.txt - log record on start

/reg/g/psdm/detector/gains/jungfrau/panels/

<repository>/logs/<year>logs/<Year>/<time-stamp>_jungfrau_deploy_constants_<login>.txt

-o, --dirrepoworks after jungfrau_dark_proc to merge and deploy constants under .../<experiment>/calib/...
jungfrau_gain_constants

>ana-4.0.44

/cds/group/psdm/logs/atstart/<year>/<year>_lcls1_jungfrau_gain_constants.txt

/reg/g/psdm/detector/gains/jungfrau/ - reference to dirs with gain constants for panel-ids


/reg/g/psdm/detectorlogs/calibcalibman/jungfrau_gain/<year>/jungfrau-<panel-id>.h5<time-stamp>-<experiment>-<run>-<login>.txt

for >ana-4.0.44:

/reg/g/psdm/

logs

detector/

calibman

gains/jungfrau

_gain

/logs/

<Year>

<year>/<time-stamp>

-

_log_jungfrau_gain_constants_<experiment>-

<run>-

r<run>_<login>.txt

-R, --repdir

DEPRECATED

jungfrau_ndarr_dark_proc



/reg/g/psdm/logs/calibman/<Year><year>/<Month><month>/jungfrau_ndarr_dark_proc.txt - log record  on record  at start
Dark processing and deployment of calibration constants for etire detector. It was DEPRECATED due to new per-panel approach to calibration constants. Replaced by jungfrau_dark_proc and jungfrau_deploy_constants

LCLS2 scripts

in repository

logs/log_epix10ka_pedestals_calibration_<Year>.txt - log record on start

TODO: move year ahead, add saving entire log under logs/<Year>/...-o, --dirrepo
scriptrecord at startrepositorylogscontrol parameterscomments
Generic calibration manager
calibman

/cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_calibman.txt

/cds/group/psdm/detector/calib2/constants/

DEPRECATED: /lcls2/cds/group/psdm/logs/calibman/lcls2/<Year>

<repository>/{scripts/<script-name>  OR <dettype>}/logs/<year>/<time-stamp>-<script-name>-<login>.txt

-L, --logdirInteraction with MongoDB etc






epix10ka
epix10ka_pedestals_calibration

/cds/group/psdm/logs/atstart/<year>/<year>_logrec_epix10ka_pedestals_calibration.txt

/cds/group/psdm/detector/calib2/constants/

DEPRECATED:

/cds/group/psdm/detector/gains2/epix10ka/panels/

<repository>/{scripts/<script-name>  OR <dettype>}/logs/<year>/

<time-stamp>-<script-name>-<login>.txt

-o, --dirrepo
epix10ka_deploy_constants

/cds/group/psdm/logs/atstart/<year>/<year>_logrec_epix10ka_deploy_constants.txt

/cds/group/psdm/detector/calib2/constants/

DEPRECATED:

/cds/group/psdm/detector/gains2/epix10ka/panels/

<repository>/{scripts/<script-name>  OR <dettype>}/logs/<year>/<time-stamp>-<script-name>-<login>.txt

-o, --dirrepo
epix10ka_charge_injection/cds/group/psdm/detector/logs/atstart/2023/2023_lcls2_epix10ka_charge_injection.txt

/cds/group/psdm/detector/calib2/constants/










epix100, opal

det_dark_proc

deploy_constants


/cds/group/psdm/detector/

gains2

logs/atstart/

epix10ka/

in repository

logs/log_epix10ka_deploy_constants_<Year>.txt - log record on start

TODO: move year ahead, add saving entire log under logs/<Year>/...

2023/2023_lcls2_det_dark_proc.txt

DEPRECATED:

/cds/group/psdm/logs/atstart/<year>/<year>_log_det_dark_proc.txt

/cds/group/psdm/detector/calib2/constants/

<repository>/{scripts/<script-name>  OR <dettype>}/logs/<year>/<time-stamp>-<script-name>-<login>.txt









Tentative logging for LCLS2

Q: is it ok to save logs in repo?

A: if repodir is changed path will be loss along with local location. It would be nice to have record at start in well known static directory. Each record points to the main log file etc.

Logging at start records

Is intended to keep all records at start for all scripts in a single place.

Root directory is intended to be hardwired, e.g. 

/cds/group/psdm/logs/atstart/

Further path

/cds/group/psdm/logs/atstart/<year>/<year>-logrec-<script-name>.txt - single file per year per script.

Repository

Root directory

-o, --dirrepo

/cds/group/psdm/detector/gains2/<detector-type>/panels/

/cds/group/psdm/detector/gains2/<detector-type>/panels/<panel-id>,

where <detector-type> e.g. epix10ka

Logging files

/cds/group/psdm/detector/gains2/<detector-type>/panels/logs/<year>/<time-stamp>-<script-name>-<login>.txt

References