Content

Introduction

In analysis of experimental data it is useful to test content of the data files. To perform this task LCLS users need in flexible interactive tool. We have evaluated a couple of currently available generic tools which present HDF5 data, HDFView and ViTables. Both of these tools have numerous pros and cons, but, definitely, as is, they can not cover all LCLS-specific graphical demands; HDFView has very primitive graphics and can not plot LCLS camera images, ViTables does not have any graphics at all, except for GUI. Extension of these packages for LCLS needs is possible, but requires significant intervention in the code of these quite specific applications. In order to keep flexibility in implementation of graphical algorithms we decided to develop our own exploring tools for XTC and HDF5 file formats. After discussion of the platform for the HDF5 Explorer we came to conclusion that most clean and compact code can be written on Python with broad exploitation of standard libraries for HDF5 access, graphics, and GUIs, such as h5py, matplotlib, and PyQt4, respectively. In this note we overview features, functionality, and status of the HDF5 Explorer. XTC Explorer - Old has its own documentation.

HDF5 Explorer features

This program is designed to access LCLS data in HDF5 files and present them in printable and graphical forms. HDF5 Explorer can perform many tasks, as follows.

• Select the input HDF5 file
• Display entire HDF5 file structure
• Print attributes and data structure of each HDF5 item (group or data set)
• Select the data-records for graphical presentation
• Select type of figures and plot
  • Image and spectra for any camera
  • Image and spectra for CSpad detector
  • X, Y, R, and Phi projections of any 2-D images
  • Waveforms
  • Correlations between scalar parameters
• Event averaging
• Apply the event selection using amplitude thresholds in specified regions of images
• Save and retrieve the configuration parameters

How to run this program

We assume that everything is set up to work on LCLS analysis farm, otherwise see Computing (including Analysis) and Account Setup. When these stages are passed successfully, the HDF5Explorer can be started on psana0### by the command

hdf5explorer

This command lunches the graphical user interface (GUI) beginning from the Main GUI. All other GUIs and plots can be lunched by clicking on mouse starting from the Main GUI.

How to get a copy and run the latest version of the HDF5Explorer?

In order to account for all users' requasts the code of HDF5Explorer may be occasionally modified. These newest modifications may not be available in current release of software. If you desperately need them, a release with a copy of the latest version of the HDF5Explorer package need to be created in your local directory. Being on psana0### one may get a source code, build executable, and run the program as shown below

cd <your-favorite-directory>
newrel ana-current myVersionOfTheHDF5Explorer
cd myVersionOfTheHDF5Explorer
sit_setup
addpkg HDF5Explorer HEAD      (at this stage you need to have access to yakut.slac.stanford.edu)
scons
hdf5explorer

GUIs

Convenient and uniform control on execution of the HDF5 Explorer program is provided by the three GUIs:

• HDF5 Explorer or Main GUI
• HDF5 tree and item selection GUI
• What to display GUI
  This section explains a functionality of the system of GUIs.

Main GUI

Main GUI is a central control unit of the HDF5 Explorer. Each button or field in this or any other HDF5 Explorer GUI can be activated by clicking on mouse left button.
The major buttons in the Main GUI are numerated in sequence of steps which need to be performed in order to plot something.
Meaning and functionality of the Main GUI buttons and fields can be listed as follows.

button opens / closes standard menu to navigate over file system and to select the HDF5 file path and name. This

to choose the data file.

The field

- is used to indicate the HDF5 file path and name. This is a fast method to change the file name, but

as an error prone due to possible typos at edition of this field.

button opens / closes the HDF5 tree and item selection GUI.

button opens / closes the What to display GUI.

button expands / collapses the Main GUI to its long / short version,

button saves the configuration parameters in the file, which is defined in the Configuration GUI.

- closes all windows and exits program. The same operation is defined for the

button in the top-right corner of the GUI window.

Expanded section of the Main GUI, consists of buttons / fields groupped in two rectangular frames.
The top frame contains the control buttons for operations with individual events in "Draw" and "Slide show" modes.
The bottom frame is intended for operations with multiple events / entries in the HDF5 data sets.
Meaning and functionality of these buttons and fields are quite intuitive as explained below.

- check box defines if the selection condition set in the Selection GUI are applied. Selection algorithm works for the "Draw", "Slide show", and "Average" modes.

- sets the current event number, which is used in "Drawing", "Slide show", and "Average" modes.

- sets the number of events for span interval. This number is used in "Drawing", "Slide show", and "Average" modes.

button resets current event and increment to 0 and 1, respectively.

- draw previous, current, or next event, respectively. In case of the "next" and "previous" button, an increment and decrement is defined by the Increment field. If the "Apply selection" check box is checked, this procedure is repeated recursively until the selection condition is fulfilled.

- buttons starts or stops the slide show, beginning from current event with interval defined by the span field. If the "Apply selection" check box is checked, this procedure shows events with fulfilled selection condition.

- this procedure loops over "number of events" beginning from current and calculate the per-event average images and waveforms. If the "Apply selection" check box is checked, this procedure accounts only for events with fulfilled selection condition. In this case the "number of events" actually means the number of selected event.

- draws the correlation plots.

- draws the calibcycle plots.

HDF5 tree and item selection GUI

HDF5 tree and item selection GUI shows the HDF5 file tree structure, which resembles the file system tree. Control buttons of this GUI are presented by the icons on tool bar and repeated in the menu bar. Their meaning and functionality from left to right on tool bar are presented as follows.

- closes the GUI window, the same as

button in the top-right corner of this window.

- saves all currently checked items for current configuration.

- resets all check boxes in unchecked state.

- restores checked items from current configuration.

- expands / collapses folders from root to the final data items.

- expands / collapses the folders for currently checked data items only.

- prints the structure of the HDF5 file tree (does not add too much to the graphical presentation of the HDF5 tree).

This GUI by itself gives an excellent opportunity to explore and check a structure of the HDF5 file. Clicking on any item of this tree, prints associated information for this item and its additional attributes, if available.

What to display GUI

What to display GUI is intended to select plots for drawing and to set specific plot parameters. Content of this GUI is generated dynamically depending on selected datasets in HDF5 tree and item selection GUI. The top part of this GUI consist of up to three checkbox sections for CSpad, Images, and Other plots. The buttom part of this GUI has a sub-GUI window, which content is selected by the tabbars. Desired plots can be selected by marking the check boxes in the top part of this GUI. When the check-mark is set, the relevant plot parameters appear in the sub-GUI. Tab bars around the Sub-GUI window also allow to switch between the parameter definition sub-GUIs. Check boxes allows to turn on/off plots as explained below.

button saves the configuration parameters in the file, which is defined in the Configuration GUI.

- closes this GUI. The same operation is defined for the

button in the top-right corner of the GUI window frame.

CSpad

This checkbox section turns on/off plots for the CSpad image datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/CsPad::ElementV2/CxiDs1.0:Cspad.0/data,
which should be checked in the HDF5 tree and item selection GUI.

- turns on/off images of 8 quad's 2x1 sensors, aligned quad image, and aligned CSpad detector image, respectively.

- turns on/off spectra of 8 quad's 2x1 sensors, 16 of separate ASICs', and entire CSPad detector.

- turns on/off image and spectrum for a single 2x1 sensor.

- turns on/off relevant projection 1-D histograms of the CSpad detector image.

Image

This checkbox section turns on/off plots for the Camera image datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/Camera::FrameV1/CxiSc1.0:Tm6740.0/image,
which should be checked in the HDF5 tree and item selection GUI.

Possible plots are:

and projections:

Other

This checkbox section turns on/off plots for other datasets;

- is intended for the "waveform" datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/Acqiris::DataDescV1/AmoETOF.0:Acqiris.0/waveforms.

- is intended for multi-parameter datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV1/CxiDg1.0:Ipimb.0/data (channel0, channel1, ...),
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV1/CxiDg1.0:Ipimb.0/time (seconds, nanoseconds), etc.

- is intended for multi-parameter datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV1/CxiDg1.0:Ipimb.0/data (channel0, channel1, ...),
/Configure:0000/Run:0000/CalibCycle:0000/Ipimb::DataV1/CxiDg1.0:Ipimb.0/time (seconds, nanoseconds), etc.

Sub-GUI window

The bottom part of the What to display GUIis intended to set parameters for particular plots trough the sub-GUI window. Content of this window can be switched by clicking on the tub bar buttons or by marking appropriate check boxes. The top tabbar buttons

switch between sub-GUIs for plot parameters . The bottom tabbar buttons

switch between the Configuration and Selection sub-GUIs.

CSpad, Image, Wave, Correlation GUIs

These sub-GUIs have more or less uniform layout of fields and buttons for monitoring and editing parameters as follows.

CSpad sub-GUI is intended to set parameters for the CSpad image and spectra plots. All CSpad plots present only one dataset, checked in the HDF5 tree and item selection GUI. (Currently, it is unlikely that users will work with more than one CSpad detector simultaneously.)

- sets the minimal and maximal amplitude for image and spectrum plots.
These amplitudes can be edited directly or set simultaneously by sliders. The slider range can also be adjusted by editing of two fields right after the

.

- sets the number of images (windows) for entire CSpad detector. Multiple images can be useful if it is necessary to zoom-in different parts of the detector simultaneously, as it is presented in the zoomed-in CSpad images.

Two radio buttons

and

- defines a primary value for the spectrum histogram, the number of bins or the bin width. The white field allows to adjust selected value, while the grey field shows the other value for monitoring purpose.

- sets the quad number in the range of [0,3] and pair number in the range of [0,7] which are used in partial CSpad plots.

Image sub-GUI is intended to set parameters for camera images and associated spectra. It has about the same set of parameters as for CSpad.
The number of datasets with images may exceed one. By default all checked in the What to display GUICamera datasets will be presented on plots separately. Otherwise the dataset name should be chosen in stead of "All" from the falling-down menu in the field

.

Waveform sub-GUI is intended to set parameters for the waveform images.

- sets the number of windows for the waveform plots.
Each window (plot) presents the waveforms for a single dataset, selected by the field

.
Each window (plot) may have up to four waveforms of different colors. Association of the waveform index and color can be done trough the buttons

.

Correlation sub-GUI is intended to set parameters for the correlation images.

- sets the number of correlation plots (windows).

... - tab bar switches between parameter sets for different plots.

Radio buttons

- switch between correlating parameter for X axis or Y-histogram for the last radio button.

- set the dataset and parameter name for Y or X axis, respectively.

check boxes turns on/off manually defined limits.

fields sets the number of bins for X and Y, respectively.

Radio buttons

and

set the scale of the Z axis for the Y vs. X correlation plot.

If for particular tabbar button the dataset is not checked in HDF5 tree and item selection GUI, the sub-GUI will advice to select the required dataset:

Calibcycle GUI

The Calibcycle GUI is identical to the Correlation GUI. The difference is that in case of "calibcycles" all parameters are defined as an average values over calibcicle. Examples of plots are shown in Calibcycle plots.

X, Y, R, and Phi projection sub-GUIs

The X, Y, R, and Phi projection sub-GUIs are intended to set parameters for the image projection plots. Currently the same set of parameters is used for CSpad detector and camera images. (We assume that user will not need in projection plots for different detectors simultaneously.)

Projection sub-GUI, selected by the button

, has its own tab bar for four type of projection plots,

These sub-GUIs have uniform layout of fields and buttons for monitoring and editing parameters for the projection plots as follows. The list of parameters for each projection plot consists of minimum and maximum value of each dimension, the number of bins or bin width for histogram along the projected dimension, and the number of slices (or rings, or sectors) and their widths for other dimension. The number of plotted 1-D histograms is equal to the number of slices for X and Y (rings for Phi, or sectors for R) projections, respectively.

For R and Phi projections it is necessary to set a center of the polar coordinates in original Cortesian coordinate frame of the image array in pixels,

Configuration GUI

tab button opens the Configuration GUI:

Configuration GUI manipulates with a set of configuration parameters.

- sets the file name for configuration parameters.

- reads configuration parameters from the file with indicated name and set them as current.

- saves current configuration parameters in the file with indicated name.

- sets current configuration parameters to their default values.

- prints current values of configuration parameters.

Selection GUI

tab button opens the Selection GUI:

Selection GUI sets parameters for the event selection algorithm based on amplitude threshold in image region(s).

- sets the number of regions (boxes) for selection algorithm. If the number of regions more than one, events are selected by logical OR of all conditions.

... - tab bar switches between parameters of different regions.

- selects the dataset with image. In case of CSpad an entire detector image is assumed.

- sets the threshold in ADC units.

Two radio buttons

and

- switch between two methods of the threshold test condition.

- sets the box range in image which is used to check the threshold condition.

Background Subtraction GUI

tab button opens the Background Subtraction GUI:

Background Subtraction GUI allows to make/set the file with background image. The checkbox turns On/Off this mode. Currently this mode works with CSPad image only.

- is an On/Off checkbox for the background subtraction mode. The check box state is not saved in the configuration file. This mode is always turned off at start of the HDF5Explorer in order to prevent user from misleading images.

- sets the file name for the background image.

- makes the background image file as a copy of the latest averaged image file.

The background subtraction file is loaded when the check box is checked. If any of involved files are not available, the error message will be printed with explanation of possible further actions.

The Background Subtraction and Gain Correction algorithms are applied to the image array in particular sequence:

        if cp.confpars.bkgdSubtractionIsOn : self.arr2dCSpad -= cp.confpars.arr_bkgd
        if cp.confpars.gainCorrectionIsOn  : self.arr2dCSpad *= cp.confpars.arr_gain

Gain Correction GUI

tab button opens the Gain Correction GUI:

Gain Correction GUI allows to make/set the file with gain correction factors for image pixels. The checkbox turns On/Off this mode. Currently this mode works with CSPad image only.

- is an On/Off checkbox for the gain correction mode. The check box state is not saved in the configuration file. This mode is always turned off at start of the HDF5Explorer in order to prevent user from misleading images.

- sets the file name for the gain correction mode.

- makes the file of the gain correction factors (f[row,col]=<Amplitude>/Amplitude[row,col]) from the latest file with averaged image.
"0"-amplitude pixels are ignored. The gain factors f<0 and f>10 are set to 0.

The gain correction file is loaded when the check box is checked. If any of involved files are not available, the error message will be printed with explanation of possible further actions.

Example of plots

Image of the CSpad detector

Run 628: Run 587:

Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in CSpad sub-GUI.

Zoomed-in CSpad image with selection window and averaged image

CSpad image can be dynamically zoomed-in. To get zoomed-in image one has to specify the window for zoomed region; click on mouse-left button in one corner of the desired region, hold it, move mouse to another corner, and release the mouse button. If you do this operation quite slow (before release button) the black dash-line rectangle will show up to indicate the selected region. Zoomed window parameters are saved for next events. Clicking on other mouse buttons in this window will restore the full-size CSpad image.
Zoomed images can be zoomed-in again using the same method.

White rectangles show the selection regions, which can be set in the Selection GUI.

The number of windows for these plots can be changed in the
CSpad sub-GUIby the

parameter.

Image of the quad

Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in CSpad sub-GUI.

Spectra of ASICs and 2x1 pairs, image and spectrum of pair

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in CSpad sub-GUI.

CSpad 2x1 sensor Image and Spectrum plot can be dynamically adjusted by clicking left/right-mouse button on color bar in order to select min/maximal amplitude for image and spectrum. Clicking on central button restores original plot parameters.

X, Y, R, and Phi projections of CSpad

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in X, Y, R, and Phi projection sub-GUIs.

Plots for camera images

Entire camera image (white box shows the selection window), zoomed-in camera image, spectrum of amplitudes and combined plot for image and spectrum

Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in Image sub-GUI.
White rectangles show the selection regions, which can be set in the Selection GUI.

Camera image can be dynamically zoomed-in (the same way like CSpad image). To get zoomed-in image one has to specify a window for zoomed region; click on mouse-left button in one corner of the desired region, hold it, move mouse to another corner, and release the button. If you do this operation quite slow (before release button) the black dash-line rectangle will show up to indicate the selected region. Zoomed window parameters are saved for next events. Clicking on other mouse buttons in this window will restore the full-size camera image.

Camera Image and Spectrum plot can be dynamically adjusted by clicking left/right-mouse button on color bar in order to select min/maximal amplitude for image and spectrum. Clicking on central button restores original plot parameters.

Camera image and X, Y, R, and Phi projections

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in X, Y, R, and Phi projection sub-GUIs.

Plots for waveforms

For auto and manual adjustment of the range and units parameters plot of waveforms are shown below:

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Wavefom sub-GUI.

Correlation plots

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Correlation sub-GUI.

Calibcycle plots

Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Calibcycle GUI.