Confluence will be unusable 23-July-2024 at 06:00 due to a Crowd upgrade.
The FFB system is designed to provide dedicated analysis capabilities during the experiment.
You can access the FFB system from pslogin, psdev or psnx with:
% ssh psffb
The experiment data will be available under:
/cds/data/drpsrcf/<instrument>/<experiment>
There are two options for telling psana which directory the data is in. One can add the "dir=" keyword to the psana DataSource, like this:
dsource = DataSource('exp=cxilu9218:run=20:smd:dir=/cds/data/drpsrcf/cxi/cxilu9218/xtc')
or one can set the following environment variable:
export SIT_PSDM_DATA=/cds/data/drpsrcf
Besides the xtc/ folder for the raw data the scratch/ folder allows user to write their processing output. This folder will be moved to the offline filesystem after an experiment is done. The calib/ is a link to the offline calib folder. A ":live" is often added to the DataSource string in order to process xtc files while they are written (currently only works for LCLS1-style DAQ/analysis).
You can submit your fast feedback analysis jobs to one of the queues shown in the following table. The goal is to assign dedicated resources to up to four experiments for each shift. Please contact your POC to get one of the high priority queues, 1, 2, 3 or 4, assigned to your experiment.
Queue | Comments | Throughput | Nodes | Cores/ | RAM [GB/node] | Default |
---|---|---|---|---|---|---|
anaq | For the week after the experiment | 100 | 62 | 61 | 128 | 12hrs |
ffbl1q | Off-shift queue for experiment 1 | 100 | 7 | 61 | 128 | 12hrs |
ffbl2q | Off-shift queue for experiment 2 | 100 | 15 | 61 | 128 | 12hrs |
ffbl3q | Off-shift queue for experiment 3 | 100 | 20 | 61 | 128 | 12hrs |
ffbl4q | Off-shift queue for experiment 4 | 100 | 25 | 61 | 128 | 12hrs |
ffbh1q | On-shift queue for experiment 1 | 100 | 7 | 61 | 128 | 12hrs |
ffbh2q | On-shift queue for experiment 2 | 100 | 15 | 61 | 128 | 12hrs |
ffbh3q | On-shift queue for experiment 3 | 100 | 20 | 61 | 128 | 12hrs |
ffbh4q | On-shift queue for experiment 4 | 100 | 25 | 61 | 128 | 12hrs |
Note that jobs submitted to ffbl<n>q will preempt jobs submitted to anaq and jobs submitted to ffbh<n>q will preempt jobs submitted to ffbl<n>q and anaq. Jobs that are preempted to make resources available to higher priority queues are paused and then are automatically resumed when resources become available.
The FFB system uses SLURM for submitting jobs - information on using SLURM can be found on the Submitting SLURM Batch Jobs page.
Once an experiment has been complete the ffb scratch folder is moved to the experiments scratch folder on the offline filesystems. The following rules are applied:
The same permission, based on ACLs, as used for the Lustre analysis file-systems are used for the FFB. However, there is an issue with the current version of the file system:
% umask 0002
The batch nodes also have access to the ana Luster filesystems. For example the calib folder /cds/data/drpsrcf/<instrument>/<experiment>/calib is a link to the folder on the ana filesystem.
The bandwidth to the ana-filesystems is very limited and must be used for only light load e.g.: reading calibration constants.
The following figure shows the connectivity of the nodes: