Page History

...

An alternative to ragged arrays are "variable" arrays.  These are not limited to 1D, but only first dimension is variable, and the other dimensions must be fixed sizes.  An HDF5 dataset name that starts with the string "var_" will generate a variable length array.  A separate integer array ending with "_len" will be automatically generated with the number of elements of the variable array that belong to each event.  (When reading such a file, a running count of the values from the "_len" array must be kept to locate the next event's data in this dataset.)

Batch Job Submission

...

Examples showing how to submit SLURM batch jobs can be found here: Submitting SLURM Batch Jobs

The following script can be a useful pattern to follow for submitting batch jobs.  Change the script to use the appropriate directory, experiment name, and analysis python-script name.   Then make the script executable with a command like "chmod +x submit.sh" and submit a job with a command like "./submit.sh 123" where "123" is the run number to be analyzed.

Code Block

#!/bin/bash myhost=`hostname` if [[ $myhost != *"psana"* ]] then echo "Need to be logged in to psana node to submit jobs" exit fi cd ~cpo/ipsana/sxrn2316 source /reg/g/psdm/etc/psconda.sh # offline system bsub -o %J.log -q psanaq -n 24 mpirun python sxrn2316.py $1 # high priority system (only to be used while experiment has beam) # bsub -o /reg/d/psdm/sxr/sxrn2316/scratch/cpo/%J.log -q psnehhiprioq -n 32 mpirun python sxrn2316.py $1

More information on the batch system can be found here.

Saving Larger Data to HDF5 ("Translation")

The MPIDataSource pattern can be used to "translate" data from xtc to hdf5.  It offers the following features:

...

TIP: If you save larger data to HDF5 (not recommended, for performance/space reasons) be sure to set the gather_interval parameter to 1 in order to avoid using up all machine memory.

References

• Batch System Analysis JobsNodes And Queues
• Detector documentation