Page History
...
- Custom bin definition: Using a run-based logic, one can define how to bin the data here. If no bins are given for a run, then the unique position of the corresponding scan motor will be used. The former is typically used for time or energy fly-scans and the latter is recommended for regular motor scans, such as phi or laser lens scans.
- Filter definition: Default filter name is
filter1
. Any other filter name will lead to the creation of separate cube files ending with_filtername
. This can be useful in case where an external field is applied on certain shots for example. - Laser on/off: If True will create
_on
and_off
cubed file. - Detectors: Area detectors dictionaries to be cubed are defined here. The options are as follow:Each detector is defined through a dictionary that can have the following keys:
thresADU
- Pixel intensity threshold
(float)
: thresADU pixel intensity threshold (float) - Common
common_mode (int/None)
: common mode selection (int/None):common_mode
.
IsNone
in most cases (take default common mode for that detector). This is generally set by your data analysis PoC and should rarely be changed. - Whether
image (bool)
: whether to return the image (True) or the calib (False, individual tiles) full (bool)
:image
Notnot-implemented (ignore for now):full
)det_proc (dict)
: a dictionary containing the name and arguments for post-processing analysis to be applied to the binned detector data. See 3.1 Post-processing functions for details on the different functions available.
- All the variable to add to the cubed file are then added to
varList
. The syntax for each non-area detector variable should follow the h5 tree structure in the smalldata h5 file. Area detectors dictionaries must be added to this list as well. - Histograms: At the end of the cube, a summary is posted to the elog. A list of variables can be customized here. The IPMs and timing tool are generally considered default. If a variable is also in the filter list, the filter boundaries will be shown in the histogram too. The pixel histograms for the area detector defined are also included in the summary. These plots can help you choose relevant filters and thresholds. Adding event codes to this list might be a good idea in cases where event code-based filters are used.
...
The cubed data are saved in a h5 files , whose fields with the following naming convention:
Cube_<exp>_<run#>_<bin_axis>_<laser_status>_<filter_name>.h5
When using laser=False
or filter1
as the filter name, the last two bits will be omitted.
The datasets are the following:
binVar_bins
: bin edges as defined in the config or the default (unique) motor positions. In the latter case, the first value is NaN, as a bin is manually added to catch outliers on the left for the general case.nEntries
: number of shots in each binbinVar
: sum of the binned variable. When binning linearly, binVar/nEntries should return the most accurate position. A field with the binned variable name contains the same information.- All the binned variables defined in
varList
. These entries are the sum of all data falling in the corresponding bin. For the regular detectors (non-area detectors) the syntax follow that of the varList, where "/
" have been replaced by "__
" (two underscores). Binned area detectors images or calib are under<detname>_data
. A<detname>_nEntries
field should match thenEntries
field. If not, contact your controls and data PoC.
The binned bin-axis variable also appears here. As for the other variables, it is summed and should be divided by nEntries to retrieve a value close to the original bin settings (if this is something of use).
Example from the config file above:ipm2__sum
ipm3__sum
diodeU__channels
jungfrau1M_data
jungfrau1M_nEntries
<detname>_cfg
: a config group for each area detector that contains the detector config (pedestals, gain map, mask, etc.)
...
Debug the cube production
The cube job can sometimes fail or get stuck. Unfortunately, the code is still in a state such that a failure won't always result in the code exiting, but instead just hanging.
Here we'll go over specific cases and how to identify them by looking at the log files.
The job seems stuck
Job is done but the report does not show "Cube: Done"
This means the job failed somewhere. Open the log files and look if one of the following cases apply:
- The most common case is that there is a typo or syntax in the config file. In this case, the log file should be very short, directly pointing at the section in the config file that has issue. As often with typo or syntax errors in python, the error is not always at the exact line mention in the error.
Check that the filters do not filter out all the pulses. In the current state, if all shots are being filtered, the job will just hang forever. This can easily spotted by looking for the following section in at the end of the log file:
Code Block language bash theme RDark did not select any event, quit now! getFilter: Cut 0.500000 < evr/code_94 < 1.500000 passes 0 events of 11252, total passes up to now: 0 getFilter: Cut 24.464119 < scan/diag_x < 24.496000 passes 11252 events of 11252, total passes up to now: 0 getFilter: Cut 0.500000 < damage/jungfrau1M < 1.500000 passes 11252 events of 11252, total passes up to now: 0
The MPI error handling is not yet in a very good state. If one of the rank fails (often rank 0), the other rank will still be hanging there, waiting for further tasks. The will this hang forever. In this, a traceback of the error will be available towards the end of the log file (it is not always at the very end). It typically can be identified by a line looking like:
Code Block language bash theme RDark Traceback (most recent call last): ...
Notes on the multi-dimensional cube
This feature is still in development, but is fully functional. Clean-up and quality of life improvements are to be expected though.
A second (or more) bin-axis can be selected following a similar run-dependent logic as for the main bin axis
Example:
Code Block | ||||
---|---|---|---|---|
| ||||
def get_addBinVars(run):
if isinstance(run,str):
run=int(run)
addBinVars = None
if run==128:
addBinVars = {'ipm2/sum': np.linspace(0,4e4,4)}
return addBinVars |
The additional axis as passed as a dictionary as {'<variable>': <bin_array>}
Cubed file content
The bin variables are handled slightly differently than in the regular cube. In the future, more uniformity can be expected.
The cubed variables now have a shape matching that of the bin-axis (i.e. (size(bin_axis1), size(bin_axis2), ...)
).
...
Analyze cube results
An example notebook of a cube file analysis can be found in /cds/group/psdm/sw/tools/smalldata_tools/example_notebooks/cube.ipynb
.