Content

Motivation

There are few tasks which need in processing of all runs, e.g. check data quality, check detector pixel status etc. This tool is intended to automatically process each data run with some pre-defined command line.

This note has a short description of entire system and references to code.

Control scripts

cron_auto_proc

Job control is performed using cron-job, currently resides in ~/bin/

...

• PSCalib/src/SubprocUtils.py

Datasets processing DB

File system based DB is used to keep info about processed runs. Everything lives under master directory, for now /reg/g/psdm/logs/run_proc

...

Run for processing finding utilities

• PSCalib/src/RunProcUtils.py

Commands

Manual launch of the scripts in stead of cron-job:

• ~/bin/cron_script_1h.sh
• ~/bin/cron_auto_proc.sh

See current processing status

• proc_new_datasets
• proc_new_datasets -s # submit jobs

Print / control DB status

• python PSCalib/src/RunProcUtils.py 1 # list of non-processed runs for experiments in the list .../pixel_status/experiments.txt
• python PSCalib/src/RunProcUtils.py 2 # list of non-processed runs for all experiments
• python PSCalib/src/RunProcUtils.py 4 # list of runs listed in logs but removed from xtc directory
• python PSCalib/src/RunProcUtils.py 5 # print statistics about all available instruments, experiments, runs.
• python PSCalib/src/RunProcUtils.py 40 # move processed runs from <experiment>-proc-runs.txt to <experiment>-arch-runs.txt for removed xtc files
• python PSCalib/src/RunProcUtils.py 10 # the same as 1, but runs are marked as processed in  .../pixel_status/<instrument>/<experiment>-proc-runs.txt
• python PSCalib/src/RunProcUtils.py 20 # the same as 2, but runs are marked as processed in  .../pixel_status/<instrument>/<experiment>-proc-runs.txt
  
  
• python PSCalib/src/SubprocUtils.py 1,2,4 #  the same as above 

References

• Detector
• PSCalib

...