Page History
Table of Contents |
---|
The small data generation takes advantage of the local development of the PSANA framework. It can be customized and run both again against the xtc files while they are being written on the ffb system (ongoing experiment only) as well as against the .xtc files on the offline system,. Parallel computing has been built-in. For a typical experiment Silke your instruments ECS engineer will set up the code in the directory directory
/reg/d/psdm/<hutch>/<expname>/results/smalldata_tools
for the offline running or /cds/data/drpsrcf/<hutch>/<experiment>/scratch/smalldata_tool
s when running in the new ffb. A "driver" python file (typically called SmallDataProducercalled
smd_producer.py
in the producers
subdirectory) can then be edited to, e.g., choose and optimize a different ROI on area detectors, define beam center for radial integration, define delay time range and bin sizes, etc. How to set up this "userData" or reduced data / feature extracted data is described in more detail in SmallData Production - DetObjectin A. Configuring smalldata production.The output will be saved to a directory that can be specified, by default this will be:
/reg/d/psdm/<hutch>/<expname>/hdf5/smalldata
or /cds/data/drpsrcf/<hutch>/<experiment>/scratch/hdf5/smalldata
The filename filenames will be of the form: <expname>_Run<runnr>.h5
To run a single DAQ run, you can use (in the res directory or in your own private release):
Setup and automatic production
smallDataRun During data taking, you can omit the "-e experiment_name" parameter and the jobs will be sent to a special queue with priority access.
smallDataRun options
129) snelson@psanaphi107:/reg/d/psdm/xpp/xpptut15/results/smalldata_tools
$ .
/examples/smallDataRun
-h
usage: .
/examples/smallDataRun
options
OPTIONS:
-r run
# (NEEDED!)
-e expname (def: current experiment, example: xppe7815)
-d directory
for
output files (default: ftc diretory of specified experiment)
-q queue name (default: psanaq
if
exp specified, psneh(hi)prioq
for
current exp)
-j
#: run on # number of cores
-n number of events (
for
testing)
-s run locally
During data taking we typically run a "procServ" process (similar to a cron job) that checks the database for the last run number and if it finds one that has not been processed yet, it will submit a job to the batch queue. The number of jobs is tuned to use as few cores as necessary to process data at data taking speed to keep the time before the files are available to a minimum while keep the the queue as empty as possible. This is useful in cases where the reduction parameters are stable for sets of runs (most of the XPP and XCS experiments fall into this category)
Overview
Content Tools