This page will describe the setup at S3DF that will be used starting in summer of '23.

Overview

Many MEC user group rely on tiff files for each detector in each event/shot taken. The necessary jobs are setup in the elog/ARP and are typically automatically triggered. A standard experiment has the following jobs:

This setup is now using smalldata_tools, where the producer is given a few options

--full: this option will translate all  the data to hdf5 files. This results in a large hdf5 files and is generally not recommended, both for disk space reasons & because this places the burden for further analysis at later steps of the analysis chain

--image: this option will store the data of tiled detectors as a single 2-d image. This is generally not recommended as this can result in discontinuities of observed features when the detector pixels are not perfectly aligned in x & Y, We usually recommend using data in raw-data shape and use the x,y&z values for each pixel in further analysis

--tiff: in addition to the hdf5 files, each dataset in 2-d shape per event will be also stored as tiff file in the scratch directory. 

Working directories

in S3DF, we have a single space for each experiment. The data can be read from two different locations: the ffb and the offline. The data is first available in the ffb and will then move towards the offline. The limited size of the ffb means that data will only be available there temporarily. As rule of thumb, use the offline if data is present there.

The results folder should hosts most of users' code, notebooks, etc. 

/cds/data/psdm/<hutch>/<experiment>/results/.

The smalldata_tools working directory generally are

/cds/data/psdm/<hutch>/<expname>/results/smalldata_tools 

for the fast feedback system (psffb).