Basic idea
The basic calibration suite is intended to provide a consistent framework for beamtime and post-beamtime monitoring, prompt analysis, and calibration. It supports all usual LCLS detectors. The scripts run transparently on psana1 and psana2 data (psana), as well as on rogue or other data generation methods that provide sufficient metadata. The code lives in git and runs on s3df. Typically we just look at raw data but the suite also supports calibrated data using internal or psana-based tools and data. Most scripts can be run single-threaded on the command line or using one or many (e.g. 100) cores in batch.
Links
The main github repo can found here: https://github.com/slaclab/beamtime-calibration-suite
It's also helpful to read through the docs found here: https://slaclab.github.io/beamtime-calibration-suite/
To gain write access to the github repo, please join the following group: https://github.com/orgs/slaclab/teams/beamtime_calibration_suite_devs
Basic setup and where to find things
To run on psana, one needs to setup conda following e.g. instructions here for psana1 or 2: S3DF stuff
To get up and running initially, follow the "Step-by-step to get running quick!" instructions from the README.txt here: https://github.com/slaclab/beamtime-calibration-suite/blob/development/README.md#step-by-step-to-get-running-quick
More setup instructions can be found on various pages of the docs, its recommended to read the following:
- Steps for initial setup (from creating github account to running example) found here: https://slaclab.github.io/beamtime-calibration-suite/setup
- Info about environment setup (using setup.sh and $OUTPUT_ROOT) found here: https://slaclab.github.io/beamtime-calibration-suite/environment
- General info about suite and directory organization found here: https://slaclab.github.io/beamtime-calibration-suite/
Generally, under beamtime-calibration-suite there are various setup scripts. For a beamtime the setup.sh script is what you want. Source that. If using psana1, export foo=1. The setup file defines $OUTPUT_ROOT, where data products will go. Under that are directories like /dark (for e.g. pedestals generated by the basic suite \[not the sort made by psana\]), /lowFlux (for single-photon studies), and /scan (timing, intensity scans).
Under beamtime-calibration-suite there are various directories but what most matters to most people is suite_scripts. One should cd there to run scripts. The scripts inherit from base classes that know about the detectors and support standard inputs. They can either run over data and produce .h5 files under $OUTPUT_ROOT/... or run on those .h5 files to produce plots and .npy files. Details about the beamline setup, such as the experiment name, the detector under study, usable modules in the camera, test pixels and regions of special interest, and the wave8 setup are controlled by a file pointed to by $SUITE_CONFIG. For non-beamtime use, there are setup files like e.g. setup_epixM_asc.sh.
General beamtime operations
At the start of most beamtimes we adjust the script configuration in $SUITE_CONFIG after getting the wave8s set up. We then take timing scan data at low and high flux, unless that's been done already. The script TimeScanParallelSlice supports this and Defay, Xavier has analysis scripts to process the output.
We then either take an official pedestal run and process/deploy it using takepeds and makepeds, or we take a plain dark run and use CalcNoiseAndMean to produce a .npy file under $OUTPUT_ROOT/dark labelled with the run number and "mean" (also a file labelled "rms"). This file can be pointed to using --fakePedestals in later scripts.
After this we tend to inspect test pixels over time using EventScanParallelSlice.py, study linearity of attenuation scans using LinearityPlotsParallelSlice.py, and evaluate the single-photon response using SimpleClustersParallelSlice.
To share hotfixes and other changes during beamtime we use github and a beamtime-specific branch we push to and pull from. More info can be found in the following section and described in the docs here: https://slaclab.github.io/beamtime-calibration-suite/workflow/
Git and GitHub usage:
git and Github are used both during development of the scripts and during beamtimes.
If you are new to git and Github, see the docs page here for some helpful links: https://slaclab.github.io/beamtime-calibration-suite/learning_git/
You can also view Stelter, Nolan's git/Github presentation and exercise here: https://drive.google.com/drive/folders/1IyltWb2rXov0kpPZ0SDdX9y9zG5h4bMl?usp=drive_link
How we use git/Github during development and beamtime is described in the docs here: https://slaclab.github.io/beamtime-calibration-suite/workflow/
And some useful git commands that might come in handy are written out here: https://slaclab.github.io/beamtime-calibration-suite/commands/
Developers:
If developing the suite, read this doc page on testing and linting the codebase: https://slaclab.github.io/beamtime-calibration-suite/testing/
Feb 2024 beamtime analysis instructions - to be reviewed/updated...
Current (Feb 2024) single photon analysis
...