This page describes smalldata_tools, a suite of code useful for analysis from the xtc data to small(er) hdf5 files at several stages of analysis. The code can be found on git at https://github.com/slac-lcls/smalldata_tools.
At XPP or XCS, the code setup is usually taken care off by the beam line staff. For other hutches, please contact the controls POC or pcds-poc-l. The working directory generally are:
/cds/data/psdm/<hutch>/<expname>/results/smalldata_tools
for the offline system (psana) and
/cds/data/drpsrcf/<hutch>/<expname>/scratch/smalldata_tools
for the (new) fast feedback system (psffb).
Two analysis infrastructures comprising of various queues and interactive nodes, are available to use depending on the status of the experiment.
Ongoing experiment are generally using the online analysis infrastructure, the fast feedback system (ffb). More info on the system here: Fast Feedback System
This system is faster and provides prioritization to ongoing experiments. Some time after the experiment is over, the access to the data will be locked and only the offline system will be available.
After the experiment is over, the data and smalldata production code are moved to the offline system, the anafs. This system available for analysis indefinitely and can be used to reprocess or refine the data.
ssh -X <ACCOUNT>@pslogin.slac.stanford.edu |
If using NoMachine, login to psnxserv.slac.stanford.edu
For the online analysis:
ssh -X psffb source /reg/g/psdm/etc/psconda.sh -py3 # Environment to use psana, etc |
And for the offline analysis:
ssh -X psana source /reg/g/psdm/etc/psconda.sh -py3 # Environment to use psana, etc |
The working directory structure can be confusing, as some of the offline folders are mounted and accessible in the online system. As a rule of thumb, until things are moved away from the online system, one should exclusively work on the ffb.
/cds/data/psdm/<hutch>/<experiment>/results/
In Jupyter hub, you can only navigate within your home folder. It is thus recommended to create shortcuts (soft-links) to the relevant experiment folders, for ease-of-access.
From Jupyter hub, click on the "+" symbol on the top left. Select "terminal" and make a soft-link to the experiment folder:
ln -s /cds/data/psdm/<hutch>/<experiment>/ ./<link>
If the experiment is going to make use of the FFB, make a second soft-link:
ln -s /cds/data/drpsrcf/<hutch>/<experiment>/ ./<link>
The data will be written to:
When using the FFB processing, data are written to:
/cds/data/drpsrcf/<hutch>/<experiment>/scratch/hdf5/smalldata
and to:
/cds/data/psdm/<hutch>/<experiment>/hdf5/smalldata
for the processing using the 'SLAC' endpoint / the psana system. Data will be moved from the FFB system to this directory within 3-4 weeks after the experiment has ended.
The analysis is generally split in two steps, allowing for easy diagnostics and customization of the analysis process. Please contact your controls and data POC to assess the best approach for your experiment.
/reg/g/psdm/sw/tools/smalldata_tools/example_notebooks
(please refrain from modifying these released notebooks in place). This approach works well for lightweight data analysis, for which the area detector images are reduced to a single (or few) number (integration of a ROI, azimuthal binning, for example) in the first step. It is also suited when detailed shot-to-shot information needs to be examined, and full control over the data binning process is desired.The contents of the smallData files are described here