This section details how to set up custom binning of the full detector images and save the output to an hdf5 file.
The main cube script is ./producers/letsCube.py
, but this file should generally not be modified. The configuration is done through a hutch-specific file of the type ./producers/cube_config_<hutch>.py
.
The configuration consists of the following main sections (see example below too):
filter1
. Any other filter name will lead to the creation of separate cube files ending with _filtername
. This can be useful in case where an external field is applied on certain shots for example._on
and _off
cubed file.thresADU (float)
common_mode
This is generally set by your data analysis PoC and should rarely be changed.image
full
varList
. The syntax for each non-area detector variable should follow the structure in the smalldata h5 file. Area detectors dictionaries must be added to this list as well.Example config file for XPP:
import numpy as np # custom bins def binBoundaries(run): if isinstance(run,str): run=int(run) if run>0: return np.arange(-5.,50.,0.2) return None # filters to apply to the data # format: list of [det (field), low, high, name] # 'filter1' is the standard name and will not be added to the h5 file name. filters = [ ['lightStatus/xray',0.5,1.5,'filter1'], ['ipm2/sum',3e2,6e4,'filter1'], ['evr/code_41',0.5,1.5,'custom'] ] # Laser on/off. laser = True # List detectors to be cubed. Area detector have additional options such as threshold # For now only full image or calib works. TODO: Add photon maps. And then any detObjectFunc # Detectors should then be added to varList detDict = {'source':'jungfrau1M', 'full':1, 'image':0, 'thresADU':6.5, 'common_mode':0} varList = ['ipm2/sum','ipm3/sum','diodeU/channels', detDict] # histogram configuration. Usually does not need to be changed # field: destination in smd, list: [low,high,n] or None (then default to some percentile) hist_list = { 'ipm2/sum': [0, 5e4, 70], 'ipm3/sum': [0, 5e3, 70], 'tt/FLTPOS_PS': [-0.5, 0.5, 70], 'tt/AMPL': [0, 0.2, 70], 'tt/FLTPOSFWHM': [0, 300, 70] } # save as tiff file (ignore unless MEC) save_tiff = False |
The cubed data are saved in a h5 files, whose fields are the following:
binVar_bins
: bin edges as defined in the config or the default (unique) motor positions. In the latter case, the first value is NaN, as a bin is manually added to catch outliers on the left for the general case.nEntries
: number of shots in each binbinVar
: sum of the binned variable. When binning linearly, binVar/nEntries should return the most accurate position. A field with the binned variable name contains the same information.varList
. These entries are the sum of all data falling in the corresponding bin. For the regular detectors (non-area detectors) the syntax follow that of the varList, where "/
" have been replaced by "__
" (two underscores). Binned area detectors images or calib are under <detname>_data
. A <detname>_nEntries
field should match the nEntries
field. If not, contact your controls and data PoC.ipm2__sum
ipm3__sum
diodeU__channels
jungfrau1M_data
jungfrau1M_nEntries
<detname>_cfg
: a config group for each area detector that contains the detector config (pedestals, gain map, mask, etc.)An example notebook that load and plots a cube file can found at: /cds/group/psdm/sw/tools/smalldata_tools/example_notebooks/cube.ipynb
Please don't modify this file directly, copy it to your home or experiment area to test and explore.
The cube job can sometimes fail or get stuck. Unfortunately, the code is still in a state such that a failure won't always result in the code exiting, but instead just hanging.
Here we'll go over specific cases and how to identify them by looking at the log files.
Check that the filters do not filter out all the pulses. In the current state, if all shots are being filtered, the job will just hang forever. This can easily spotted by looking for the following section in the log file:
did not select any event, quit now! getFilter: Cut 0.500000 < evr/code_94 < 1.500000 passes 0 events of 11252, total passes up to now: 0 getFilter: Cut 24.464119 < scan/diag_x < 24.496000 passes 11252 events of 11252, total passes up to now: 0 getFilter: Cut 0.500000 < damage/jungfrau1M < 1.500000 passes 11252 events of 11252, total passes up to now: 0 |
The MPI error handling is not yet in a very good state. If one of the rank fails (often rank 0), the other rank will still be hanging there, waiting for further tasks. The job will hang forever. In this case, a traceback of the error will be printed towards the end of the log file (it is not always at the very end). It typically can be identified by a line looking like:
Traceback (most recent call last): ... |
An example notebook of a cube file analysis can be found in /cds/group/psdm/sw/tools/smalldata_tools/example_notebooks/cube.ipynb
.