In analysis of experimental data it is useful to test content of the data files. To perform this task LCLS users need in flexible interactive tool. We have evaluated a couple of currently available generic tools which present HDF5 data, HDFView and ViTables. Both of these tools have numerous pros and cons, but, definitely, as is, they can not cover all LCLS-specific graphical demands; HDFView has very primitive graphics and can not plot LCLS camera images, ViTables does not have any graphics at all, except for GUI. Extension of these packages for LCLS needs is possible, but requires significant intervention in the code of these quite specific applications. In order to keep flexibility in implementation of graphical algorithms we decided to develop our own exploring tools for XTC and HDF5 file formats. After discussion of the platform for the HDF5 Explorer we came to conclusion that most clean and compact code can be written on Python with broad exploitation of standard libraries for HDF5 access, graphics, and GUIs, such as h5py, matplotlib, and PyQt4, respectively. In this note we overview features, functionality, and status of the HDF5 Explorer. XTC Explorer has its own documentation.
This program is designed to access LCLS data in HDF5 files and present them in printable and graphical forms. HDF5 Explorer can perform many tasks, as follows.
We assume that everything is set up to work on LCLS analysis farm, otherwise see
and Account Setup. When these stages are passed successfully, the HDF5Explorer can be started on psana0### by the commandhdf5explorer |
This command lunches the graphical user interface (GUI) beginning from the Main GUI. All other GUIs and plots can be lunched by clicking on mouse starting from the Main GUI.
If you are going to use the HDF5 files from Lustre file system, this program should be run on one of the psana0### computers. |
How to get a copy and run the latest version of the HDF5Explorer?
In order to account for all users' requasts the code of HDF5Explorer may be occasionally modified. These newest modifications may not be available in current release of software. If you desperately need them, a release with a copy of the latest version of the HDF5Explorer package need to be created in your local directory. Being on psana0### one may get a source code, build executable, and run the program as shown below
cd <your-favorite-directory> newrel ana-current myVersionOfTheHDF5Explorer cd myVersionOfTheHDF5Explorer sit_setup addpkg HDF5Explorer HEAD (at this stage you need to have access to yakut.slac.stanford.edu) scons hdf5explorer |
Convenient and uniform control on execution of the HDF5 Explorer program is provided by the three GUIs:
Main GUI is a central control unit of the HDF5 Explorer. Each button or field in this or any other HDF5 Explorer GUI can be activated by clicking on mouse left button.
The major buttons in the Main GUI are numerated in sequence of steps which need to be performed in order to plot something.
Meaning and functionality of the Main GUI buttons and fields can be listed as follows.
button opens / closes standard menu to navigate over file system and to select the HDF5 file path and name. This
{highlight:#ccffcc}is a recommended way{highlight} |
The field
{highlight:#dddddd} {highlight} |
- is used to indicate the HDF5 file path and name. This is a fast method to change the file name, but
{highlight:#ffccff}this is not a recommended way{highlight} |
button opens / closes the HDF5 tree and item selection GUI.
button opens / closes the What to display GUI.
button expands / collapses the Main GUI to its long / short version,
button saves the configuration parameters in the file, which is defined in the Configuration GUI.
- closes all windows and exits program. The same operation is defined for the
button in the top-right corner of the GUI window.
Expanded section of the Main GUI, consists of buttons / fields groupped in two rectangular frames.
The top frame contains the control buttons for operations with individual events in "Draw" and "Slide show" modes.
The bottom frame is intended for operations with multiple events / entries in the HDF5 data sets.
Meaning and functionality of these buttons and fields are quite intuitive as explained below.
{highlight:#dddddd} Apply selection {highlight} |
{highlight:#dddddd} Current event: {highlight} |
- sets the current event number, which is used in "Drawing", "Slide show", and "Average" modes.
{highlight:#dddddd} Increment: {highlight} |
- sets the number of events for span interval. This number is used in "Drawing", "Slide show", and "Average" modes.
button resets current event and increment to 0 and 1, respectively.
{highlight:#dddddd} Draw: {html}<INPUT type="button" value="Previous"></INPUT> <INPUT type="button" value="Current"></INPUT> <INPUT type="button" value="Next"></INPUT>{html}{highlight} |
{highlight:#dddddd} Slide show: {html}<INPUT type="button" value="Start"></INPUT> <INPUT type="button" value="Stop"></INPUT>{html}{highlight} |
{highlight:#dddddd}{html}<INPUT mce_style="background-color: #eeffee;" style="background-color: rgb(230, 255, 230);" type="button" value="Average"></INPUT>{html} over {html}<INPUT align="middle" size="14" type="text" value="number of events"></INPUT>{html} events {highlight} |
- draws the correlation plots.
- draws the calibcycle plots.
HDF5 tree and item selection GUI shows the HDF5 file tree structure, which resembles the file system tree. Control buttons of this GUI are presented by the icons on tool bar and repeated in the menu bar. Their meaning and functionality from left to right on tool bar are presented as follows.
{highlight:#dddddd} Exit {highlight} |
button in the top-right corner of this window.
{highlight:#dddddd} Check {highlight} |
{highlight:#dddddd} Reset {highlight} |
{highlight:#dddddd} Retrieve {highlight} |
{highlight:#dddddd} Expand / collapse {highlight} |
{highlight:#dddddd} Expand / collapse checked {highlight} |
{highlight:#dddddd} Print {highlight} |
This GUI by itself gives an excellent opportunity to explore and check a structure of the HDF5 file. Clicking on any item of this tree, prints associated information for this item and its additional attributes, if available.
All datasets, which you plan to use in graphical presentations, have to be checked in their individual check boxes. When a desired list of datasets is chosen, it needs to be passed to the current configuration by clicking on the
|
What to display GUI is intended to select plots for drawing and to set specific plot parameters. Content of this GUI is generated dynamically depending on selected datasets in HDF5 tree and item selection GUI. The top part of this GUI consist of up to three checkbox sections for CSpad, Images, and Other plots. The buttom part of this GUI has a sub-GUI window, which content is selected by the tabbars. Desired plots can be selected by marking the check boxes in the top part of this GUI. When the check-mark is set, the relevant plot parameters appear in the sub-GUI. Tab bars around the Sub-GUI window also allow to switch between the parameter definition sub-GUIs. Check boxes allows to turn on/off plots as explained below.
button saves the configuration parameters in the file, which is defined in the Configuration GUI.
- closes this GUI. The same operation is defined for the
button in the top-right corner of the GUI window frame.
This checkbox section turns on/off plots for the CSpad image datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/CsPad::ElementV2/CxiDs1.0:Cspad.0/data,
which should be checked in the HDF5 tree and item selection GUI.
{highlight:#dddddd} Images: {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [8 of 2x1 |PCDS:HDF5 Event Display# Spectra of ASICs and 2x1 pairs, image and spectrum of pair] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Quad|PCDS:HDF5 Explorer# Image of the quad] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Detector |PCDS:HDF5 Explorer# Image of the detector] {highlight} |
{highlight:#dddddd} Spectra: {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [8 of 2x1|PCDS:HDF5 Event Display#Spectra of ASICs and 2x1 pairs, image and spectrum of pair] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [16 ASICs|PCDS:HDF5 Explorer#Spectra of ASICs and 2x1 pairs, image and spectrum of pair] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} Detector {highlight} |
{highlight:#dddddd} Image & Spectrum: {html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[1 of 2x1|PCDS:HDF5 Explorer#Spectra of ASICs and 2x1 pairs, image and spectrum of pair] {highlight} |
{highlight:#dddddd} Projections: {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [X |PCDS:HDF5 Explorer#X, Y, R, and Phi projections of CSpad] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Y |PCDS:HDF5 Explorer#X, Y, R, and Phi projections of CSpad] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [R |PCDS:HDF5 Explorer#X, Y, R, and Phi projections of CSpad] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Phi|PCDS:HDF5 Explorer#X, Y, R, and Phi projections of CSpad] {highlight} |
For Mini CSPad (a.k.a. 2x2), consisting of two 2x1 sensors, the plots like
The
|
This checkbox section turns on/off plots for the Camera image datasets, for example
/Configure:0000/Run:0000/CalibCycle:0000/Camera::FrameV1/CxiSc1.0:Tm6740.0/image,
which should be checked in the HDF5 tree and item selection GUI.
Possible plots are:
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[Image |PCDS:HDF5 Explorer#Plots for camera images] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Spectrum |PCDS:HDF5 Explorer#Plots for camera images] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[Image and Spectrum |PCDS:HDF5 Explorer#Plots for camera images] {highlight} |
and projections:
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [X |PCDS:HDF5 Explorer#Camera image and X, Y, R, and Phi projections] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Y |PCDS:HDF5 Explorer#Camera image and X, Y, R, and Phi projections] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [R |PCDS:HDF5 Explorer#Camera image and X, Y, R, and Phi projections] {html}<INPUT checked="checked" type="checkbox"></INPUT>{html} [Phi|PCDS:HDF5 Explorer#Camera image and X, Y, R, and Phi projections] {highlight} |
This checkbox section turns on/off plots for other datasets;
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[Waveform|PCDS:HDF5 Explorer#Plots for waveforms] {highlight} |
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[Correlations|PCDS:HDF5 Explorer#Correlation plots] {highlight} |
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html}[CalibCycles|PCDS:HDF5 Explorer#Calibcycle plots] {highlight} |
The bottom part of the What to display GUI is intended to set parameters for particular plots trough the sub-GUI window. Content of this window can be switched by clicking on the tub bar buttons or by marking appropriate check boxes. The top tabbar buttons
{highlight:#dddddd}{html}<INPUT mce_style="color:red" style="color: red;" type="button" value="CSpad"></INPUT> <INPUT mce_style="color:blue" style="color: blue;" type="button" value="Image"></INPUT> <INPUT mce_style="color:green" style="color: green;" type="button" value="Waveform"></INPUT> <INPUT mce_style="color:magenta" style="color: magenta;" type="button" value="Proj."></INPUT> <INPUT mce_style="color:black" style="color: black;" type="button" value="Corr."></INPUT>{html} {highlight} |
{highlight:#dddddd}{html}<INPUT mce_style="color:black" style="color: black;" type="button" value="Configuration"></INPUT><INPUT mce_style="color:black" style="color: black;" type="button" value="Selection"></INPUT>{html} {highlight} |
These sub-GUIs have more or less uniform layout of fields and buttons for monitoring and editing parameters as follows.
CSpad sub-GUI is intended to set parameters for the CSpad image and spectra plots. All CSpad plots present only one dataset, checked in the HDF5 tree and item selection GUI. (Currently, it is unlikely that users will work with more than one CSpad detector simultaneously.)
{highlight:#dddddd} Image plot / Spectrum Amin: {html}<INPUT align="middle" size="4" type="text" value="0"></INPUT>{html} Amax: {html}<INPUT align="middle" size="4" type="text" value="1000"></INPUT>{html}{highlight} |
{highlight:#dddddd} Slider range:{highlight} |
{highlight:#dddddd} N detector images : {highlight} |
- sets the number of images (windows) for entire CSpad detector. Multiple images can be useful if it is necessary to zoom-in different parts of the detector simultaneously, as it is presented in the zoomed-in CSpad images.
Two radio buttons
{highlight:#dddddd}{html}<INPUT checked="checked" type="radio"></INPUT>{html} Bin width: {highlight} |
and
{highlight:#dddddd}{html}<INPUT type="radio"></INPUT>{html} N bins: {highlight} |
- defines a primary value for the spectrum histogram, the number of bins or the bin width. The white field allows to adjust selected value, while the grey field shows the other value for monitoring purpose.
{highlight:#dddddd} Quad: {html}<INPUT type="button" value="1?"></INPUT>{html} Pair: {html}<INPUT type="button" value="0?"></INPUT>{html}{highlight} |
Image sub-GUI is intended to set parameters for camera images and associated spectra. It has about the same set of parameters as for CSpad.
The number of datasets with images may exceed one. By default all checked in the What to display GUI Camera datasets will be presented on plots separately. Otherwise the dataset name should be chosen in stead of "All" from the falling-down menu in the field
{highlight:#dddddd} Dataset: {html}<INPUT size="30" type="button" value="All"></INPUT>{html}{highlight} |
Waveform sub-GUI is intended to set parameters for the waveform images.
{highlight:#dddddd} Number of windows : {highlight} |
- sets the number of windows for the waveform plots.
Each window (plot) presents the waveforms for a single dataset, selected by the field
{highlight:#dddddd} Dataset: {html}<INPUT size="30" type="button" value="Click here to select the dataset"></INPUT>{html}{highlight} |
{highlight:#dddddd} WF: Black {html}<INPUT type="button" value="0"></INPUT>{html} Red {html}<INPUT type="button" value="1"></INPUT>{html} Green {html}<INPUT type="button" value="3"></INPUT>{html} Blue {html}<INPUT type="button" value="None"></INPUT>{html}{highlight} |
Correlation sub-GUI is intended to set parameters for the correlation images.
{highlight:#dddddd} Number of correlation plots: {highlight} |
- sets the number of correlation plots (windows).
... - tab bar switches between parameter sets for different plots.
Radio buttons
{highlight:#dddddd}{html}<INPUT checked="checked" type="radio"></INPUT>{html} Index {html}<INPUT type="radio"></INPUT>{html} Time {html}<INPUT type="radio"></INPUT>{html} X-par {html}<INPUT type="radio"></INPUT>{html} Y-hist. {highlight} |
{highlight:#dddddd} Y/X-par: {html}<INPUT type="button" value="dataset name"></INPUT><INPUT type="button" value="parameter name"></INPUT>{html}{highlight} |
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html} X/Ylims: {highlight} |
{highlight:#dddddd} X/Y Nbins: {highlight} |
Radio buttons
{highlight:#dddddd}{html}<INPUT type="radio"></INPUT>{html} log Z{highlight} |
{highlight:#dddddd}{html}<INPUT checked="checked" type="radio"></INPUT>{html} lin Z{highlight} |
If for particular tabbar button the dataset is not checked in HDF5 tree and item selection GUI, the sub-GUI will advice to select the required dataset:
The Calibcycle GUI is identical to the Correlation GUI. The difference is that in case of "calibcycles" all parameters are defined as an average values over calibcicle. Examples of plots are shown in Calibcycle plots.
The X, Y, R, and Phi projection sub-GUIs are intended to set parameters for the image projection plots. Currently the same set of parameters is used for CSpad detector and camera images. (We assume that user will not need in projection plots for different detectors simultaneously.)
Projection sub-GUI, selected by the button
, has its own tab bar for four type of projection plots,
{highlight:#dddddd} {html}<INPUT mce_style="color:red" style="color: red;" type="button" value="X"></INPUT> <INPUT mce_style="color:magenta" style="color: magenta;" type="button" value="Y"></INPUT> <INPUT mce_style="color:blue" style="color: blue;" type="button" value="R"></INPUT> <INPUT mce_style="color:green" style="color: green;" type="button" value="Phi"></INPUT>{html} {highlight} |
These sub-GUIs have uniform layout of fields and buttons for monitoring and editing parameters for the projection plots as follows. The list of parameters for each projection plot consists of minimum and maximum value of each dimension, the number of bins or bin width for histogram along the projected dimension, and the number of slices (or rings, or sectors) and their widths for other dimension. The number of plotted 1-D histograms is equal to the number of slices for X and Y (rings for Phi, or sectors for R) projections, respectively.
For R and Phi projections it is necessary to set a center of the polar coordinates in original Cortesian coordinate frame of the image array in pixels,
{highlight:#dddddd} Center X and Y {html}<INPUT size="3" type="text" value="850"></INPUT><INPUT size="3" type="text" value="850"></INPUT>{html}{highlight} |
tab button opens the Configuration GUI:
Configuration GUI manipulates with a set of configuration parameters.
- sets the file name for configuration parameters.
- reads configuration parameters from the file with indicated name and set them as current.
- saves current configuration parameters in the file with indicated name.
- sets current configuration parameters to their default values.
- prints current values of configuration parameters.
tab button opens the Selection GUI:
Selection GUI sets parameters for the event selection algorithm based on amplitude threshold in image region(s).
{highlight:#dddddd} Number of regions for selection : {highlight} |
- sets the number of regions (boxes) for selection algorithm. If the number of regions more than one, events are selected by logical OR of all conditions.
... - tab bar switches between parameters of different regions.
{highlight:#dddddd} Dataset : {html}<INPUT type="button" value="dataset name"></INPUT>{html}{highlight} |
{highlight:#dddddd} Threshold on intensity : {html}<INPUT size="3" type="text" value="value"></INPUT>{html}{highlight} |
Two radio buttons
{highlight:#dddddd}{html}<INPUT checked="checked" type="radio"></INPUT>{html} maximal {highlight} |
{highlight:#dddddd}{html}<INPUT type="radio"></INPUT>{html} integral {highlight} |
{highlight:#dddddd} X/Ymin, X/Ymax: {html}<INPUT size="3" type="text" value="1"></INPUT> <INPUT size="3" type="text" value="1111"></INPUT>{html}{highlight} |
tab button opens the Background Subtraction GUI:
Background Subtraction GUI allows to make/set the file with background image. The checkbox turns On/Off this mode. Currently this mode works with CSPad image only.
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html} Apply background subtraction{highlight} |
- sets the file name for the background image.
- makes the background image file as a copy of the latest averaged image file.
The background subtraction file is loaded when the check box is checked. If any of involved files are not available, the error message will be printed with explanation of possible further actions.
The Background Subtraction and Gain Correction algorithms are applied to the image array in particular sequence:
if cp.confpars.bkgdSubtractionIsOn : self.arr2dCSpad -= cp.confpars.arr_bkgd if cp.confpars.gainCorrectionIsOn : self.arr2dCSpad *= cp.confpars.arr_gain |
tab button opens the Gain Correction GUI:
Gain Correction GUI allows to make/set the file with gain correction factors for image pixels. The checkbox turns On/Off this mode. Currently this mode works with CSPad image only.
{highlight:#dddddd}{html}<INPUT checked="checked" type="checkbox"></INPUT>{html} Apply gain correction{highlight} |
- sets the file name for the gain correction mode.
- makes the file of the gain correction factors (f[row,col]=<Amplitude>/Amplitude[row,col]) from the latest file with averaged image.
"0"-amplitude pixels are ignored. The gain factors f<0 and f>10 are set to 0.
The gain correction file is loaded when the check box is checked. If any of involved files are not available, the error message will be printed with explanation of possible further actions.
Run 628: Run 587:
Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in CSpad sub-GUI.
CSpad image can be dynamically zoomed-in. To get zoomed-in image one has to specify the window for zoomed region; click on mouse-left button in one corner of the desired region, hold it, move mouse to another corner, and release the mouse button. If you do this operation quite slow (before release button) the black dash-line rectangle will show up to indicate the selected region. Zoomed window parameters are saved for next events. Clicking on other mouse buttons in this window will restore the full-size CSpad image.
Zoomed images can be zoomed-in again using the same method.
White rectangles show the selection regions, which can be set in the Selection GUI.
The number of windows for these plots can be changed in the
CSpad sub-GUI by the
{highlight:#dddddd} N detector images : {highlight} |
parameter.
Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in CSpad sub-GUI.
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in CSpad sub-GUI.
CSpad 2x1 sensor Image and Spectrum plot can be dynamically adjusted by clicking left/right-mouse button on color bar in order to select min/maximal amplitude for image and spectrum. Clicking on central button restores original plot parameters.
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in X, Y, R, and Phi projection sub-GUIs.
Entire camera image (white box shows the selection window), zoomed-in camera image, spectrum of amplitudes and combined plot for image and spectrum
Plots can be turned on/off in What to display GUI.
Parameters for this plot can be set in Image sub-GUI.
White rectangles show the selection regions, which can be set in the Selection GUI.
Camera image can be dynamically zoomed-in (the same way like CSpad image). To get zoomed-in image one has to specify a window for zoomed region; click on mouse-left button in one corner of the desired region, hold it, move mouse to another corner, and release the button. If you do this operation quite slow (before release button) the black dash-line rectangle will show up to indicate the selected region. Zoomed window parameters are saved for next events. Clicking on other mouse buttons in this window will restore the full-size camera image.
Camera Image and Spectrum plot can be dynamically adjusted by clicking left/right-mouse button on color bar in order to select min/maximal amplitude for image and spectrum. Clicking on central button restores original plot parameters.
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in X, Y, R, and Phi projection sub-GUIs.
For auto and manual adjustment of the range and units parameters plot of waveforms are shown below:
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Wavefom sub-GUI.
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Correlation sub-GUI.
Plots can be turned on/off in What to display GUI.
Parameters for these plots can be set in Calibcycle GUI.