Page History
Table of Contents | ||
---|---|---|
|
...
xtcexplorer (GUI interface to pyana)
This document describes XtcExplorer The latest tag is V00-01-1631.
This is a description and documentation of a GUI tool to explore the xtc data. It XTC data (see Analysis Workbook. Data Formats). The executable is named xtcexplorer, and the software package is named XtcExplorer. The GUI is written in python, relying on PyQt4 for graphical user interface. The data processing is done via the pyana framework and visualization provided by matplotlib. It can be run on any xtc file, including the file being written during data aquisition (DAQ). Python/PyQt, while the data is extracted using the LCLS framework pyana (we're currently in process of switching to using psana for this). Visualization of data is obtaned with the plotting package matplotlib, while much of the data algorithms in python relies on NumPy / SciPy libraries.
XTC Xtc files contain the raw data (datagrams) streamed from the DAQ online system. The datagrams occur in the order they were flushed from the DAQ system, and thus do not necessarily appear in chronological order. The datagrams are not indexed, so the file must be read sequentially. The pyana and psana frameworks are designed to process these datagrams and present them to you shot by shot (or event by event, in pyana/psana terminology).
The XtcExplorer
allows a simple-to-run interface to the xtc files. The package name is XtcExplorer, the executables (xtcexplorer and xtcscanner) are found in the app subdirectory of this package, and all other source code is in the src subdirectory.
For a similar tool to explore the HDF5 data format, look at HDF5 Explorer - Old.
How to get started
Set up your release work directory:
the section
You should always run this code from your own release working directory.
Seefor more details on how to set up the analysis environment
. Set up your release working directory frompslogin
::
AddCode Block none none [user@pslogin ~] newrel ana-current myrelease [user@pslogin ~] cd myrelease [user@pslogin myrelease] sit_setup
Check out the
Since release ana-0.3.7, xtcexplorer has been built into the release so you can skip to (3).XtcExplorer
package from the SVN software repository:
This step is optional.
Checking out a local copy is beneficial, though! That will enable you to edit the source files and customise the analysisBut without it, you will not be able to edit/customize any source files.
, so
You need an afs token to check out the package, which on
didn't already you need to run 'kinit'pslogin
is automatic (I think), but if you're on psananeh/psanafeh, issue a
kinit
command first:Code Block none none [user@pslogin myrelease] kinit Password for user@SLAC.STANFORD.EDU:
Now you're ready to check out the package and "compile":
Code Block none none [user@pslogin myrelease] addpkg XtcExplorer [user@pslogin myrelease] scons
Note! You can also do '
to get the most recent one, doaddpkg XtcExplorer VXX-XX-XX
' to get a specific version of the package, or'
, but be aware that it may look different from described here, and might have fresh bugs as well as new featuresaddpkg XtcExplorer HEAD
'to get the very latest version.
Run the program:
optionally
Launch the GUI with the command 'xtcexplorer' andoptionaly give the input xtc files that you want to read
as arguments. You can also browse to find files after launching the browser.
To have access to the xtc files, you need to login topsana
psananeh/psanafeh
for this:Code Block [user@psanaXXX myrelease] sit_setup [user@psanaXXX myrelease] xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0*
Info |
---|
|
Description of the GUIs
The following documentation describes version 0.1.15 (or "tag" XtcExplorer V00-01-15")31. For earlier versions, see earlier versions of this confluence page (go to "Tools" -> "Page history").
LCLS
...
XTC Explorer
Typing xtcexplorer will launch the main GUI. At startup, it has two sections:
- File section: Shows a list of currently selected file(s). Please be adviced that you should only select a single run at a time (or combine runs where you know the setup / configuration is the same). A single run is typically spread over several XTC files (divided into streams (-s0X) and chunks (-c0X)).
A "Clear File List" removes all filenames from the list.
You have three options for adding files:- You can supply file names at the command line when launching the application.
- A "File Browser" button opens a file browser, where you can select one or more files.
- Or you can type file name by hand in the input line.
...
, which looks something like this:
You need to connect an XTC file in order to get any further. You have 4 options:
- You can give file names as arguments on the command line, and the file will automatically be opened/connected< e.g.
xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0*
- If you know the Instrument / Experiment number / run number you'd like to investigate, simply select from the pull-down menus.
- There's also a 'File Browser' that allows you to browse for the files you want. Be aware that many runs are split into several files, in chunks and streams, and to get a complete dataset you do need to use all of the files for a given run. It will probably 'work' for single files too, but we cannot guarantee completely synchronized date, in particular, EPICS data is stored at a different rate than 'event' (shot) data, so if you make use of EPICS PV data, use the complete dataset.
- Finally, you have the option of adding file names by hand.
Once a file or more has been connected, another GUI will pop up (possibly after a short delay while the file(s) content is being investigated). This GUI displays information about the file content and help with further processing of the data in pyana.
...
...
The main window before any file selection. You can select files either by editing the filename by hand or by opening the "File Browser...". If you select instrument and experiment the browser will take you directly to the right directory, or if you type in the run number it will load all files for this run number.
...
...
A standard file browser pops up. Navigate the file browser till you find the xtc files for your experiment. (Remember, if this seems cumbersome, you can also give the file name(s) as argument to xtcexplorer when you first launch the GUI)
...
...
Mark the files you want to look at (Hold down the Ctrl-key to select more than one file). Note, you should only look at one run number at a time, since different runs might have different configurations.
...
...
This is the main window after a file has been selected. File name and file size is shown in the GUI. If the file is not too big, you can click the "Scan File(s)" button to get exact contents of the whole file. If the file is big, it's better to do a "Quick Scan" which will tell you all you need to know (except count number of events and calibration cycles).
Pyana Control Center
After scanning, a new GUI will pop up showing you a list of detectors/devices found in the file. A little more information is written to the terminal window too.
...
...
Main window after the file scan. Displays the number of events and calibration cycles (scan steps)
...
...
...
Some more information is printed to the terminal window from which the xtcbrowser was launched
Pyana Control Center
The new GUI (Pyana Control Center) will show you a list of detectors/devices found in the file. A little more information is written to the terminal window too.
The Pyana Control Center. | |
Some more information is printed to the terminal window from which the xtcexplorer was launched | |
| Once you checkmark the detectors you want to display information from, another tab will pop up showing pyana configuration text. "Current pyana configuration": as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field. |
More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).
The pyana modules
The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!
Code Block | ||||
---|---|---|---|---|
| ||||
XtcExplorer/src/pyana_bld.py # display of Beam-line data
XtcExplorer/src/pyana_epics.py # display of Epics PV data
XtcExplorer/src/pyana_scan.py # display of motor scan data
XtcExplorer/src/pyana_encoder.py # display of encoder data
XtcExplorer/src/pyana_ipimb.py # display of diode data from IPIMB and PIM
XtcExplorer/src/pyana_waveform.py # display of waveform data
XtcExplorer/src/pyana_image.py # display of camera image data
XtcExplorer/src/pyana_plotter.py # a plotter module to control the event display
|
Additional helper classes and functions are defined in
Code Block | ||||
---|---|---|---|---|
| ||||
XtcExplorer/src/utilities |
...
...
More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).
The pyana modules
The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!
Code Block | ||
---|---|---|
none | none | XtcExplorer/src/pyana_bld.py # displayPyanaOptions of Beam-line data XtcExplorer/src/pyana_epics.py # display of Epics PV data XtcExplorer/src/pyana_scan.py # display of motor scan data XtcExplorer/src/pyana_encoder.py - help parse options # display of encoder data XtcExplorer/src/pyana_ipimb.py # display of diode data from IPIMB and PIM XtcExplorer/src/pyana_waveform.py # display of waveform data XtcExplorer/src/pyana_image.py # BaseData #and displayclasses ofderived camerathereof image data XtcExplorer/src/pyana_plotter.py - store #numpy aarrays plotterfor moduleipython to control the event display |
Additional helper classes and functions are defined in
Code Block | ||
---|---|---|
none | none | XtcExplorer/src/utilities.py # PyanaOptions # Threshold - help parse options - keep track of threshold values and region of interest # BaseData and classes derived thereof - store numpy arrays for ipython # Frame # Threshold - container for a single "plot" - keep track of threshold values and region of interest # Frame Plotter - container for a single "plotcanvas" XtcExplorer/src/cspad.py # CsPad # Plotter - container for aarranging single "canvas" XtcExplorer/src/cspad.py # CsPad - for arranging & aligning the CsPad elements & aligning the CsPad elements |
These These are imported into the pyana modules wherever needed. You might want to keep this in mind when
using examples from this package. The pyana modules can mostly be run on their own, but they might
need to import this module.
...
A few things to note about the different detectors / pyana modules:
Common configuration parameters
A few parameters are common to all modules, and their default values are set to be the same, but can also be set individually for each module. They are the following:
Code Block | ||||
---|---|---|---|---|
| ||||
# parameter name default value comment
plot_every_n = 100 # If 0, plot only at the end. Else, display every N events
accumulate_n = 0 # If 0, accumulate all (no reset). Else, reset stored arrays every N events
fignum = 1 # matplotlib figure number
|
pyana_plotter
This module is added to the end of the job, after other modules. It controls the display mode and ipython.
- The default display mode is SlideShow mode, where the plot window stays open but updates every n events. In Interactive mode, the figure window must be closed before the next event gets processed. The other modules are set up to work with this one, but should with minor modifications also run fine without it. (The minor modifications would be to issue a "plt.ion()" before the figures are created or replace "plt.draw()" with "plt.show()" to make sure the plot windows stay open.)
There's also a handle to allow you to use ipython at the end of the job (or after each plot) to investigate your plottet arrays further.
Code Block none none display_mode = 2 # Interactive (1) or SlideShow (2) or NoDisplay (0) ipython = False # Drop into ipython after each plot (to allow further investigation of data)
pyana_bld
This module collects data from the BeamLine Detectors (Bld) and plots them in separate windows.
Code Block | ||||
---|---|---|---|---|
| ||||
do_ebeam = False # Plot data from EBeam object
do_gasdetector = False # Plot data from GasDetector
do_phasecavity = False # Plot data from PhaseCavity
do_ipimb = False # Plot data from SharedIpimb (No longer used, see pyana_ipimb)
|
...
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_bld -m XtcExplorer.pyana_plotter <xtc files>
pyana_waveform
Plots waveform data from Acqiris digitizers. Takes a list of addresses, currently will make one plot for each channel of each device. Plan to add more options.
Configurable options:
Code Block | ||||
---|---|---|---|---|
| ||||
sources = # string, space-separated list of addresses (Detector-Id|Device-ID)
quantities = # string containing quantities to plot. "average stack single"
|
...
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform <xtc files>
pyana_image
This module will take care of displaying deals with all camera images, including CsPad images. Additionally, it can
- Load dark image from a file for background subtraction
- Filter events: the 'threshold' parameter holds settings for filtering events based on image values. Specify keywords=value separated by whitespaces, e.g.
- 'lower=400.0 upper=1600.0 type=maximum roi=(200:300,100:400)' which will filter out events that have the maximum pixel in the region of interest below 400 or above 1600.
- 'lower=500 type=average' will filter out events that have average pixel value in the whole image below 500.
- Apply algorithms to an image (e.g. rotation, shift, etc)
- Display and/or save the image, a background subtacted image, an average image, an image with algorithms applied... etc.
- filter images above a threshold value
- filter images with ROI above a threshold value
- apply algorithms to an image (e.g. rotation, shift, etc)
- Sum&average images above threshold, and images below threshold
- Load dark image from a file for background subtraction
- Save the average image(s) or selected single-shot images (up to a maximum) to file (numpy binary file)
- In Interactive mode (see pyana_plotter), some interaction with the plot is possible:
- change intensity range (color range) by left-clicking to set low limit, right-clicking to set high limit, middle-click to reset to full scale.
- change threshold value and region of interest if a ROI was defined: Click on the rectangle, enter new values when prompted in terminal window
- switch to SlideShow mode when you are content with the display: Middle-click on the canvas and enter new display mode value (2 for SlideShow).
Options must be specified in a configuration file. Only the sources parameter is required.
Code Block | ||||
---|---|---|---|---|
| ||||
sources = # string, Address of Detector-Id|Device-ID inputdark = # filename of a binary numpy (.npy) with dark image for subtraction threshold = # Threshold information, format: 'value (specify the keywords that apply: lower=value, upper=value, type=maximum (default) or average, roi=(xmin:xmax,ymin:xmaxymax) type' algorithms = # algorithms to be applied to the image, e.g.: 'rotate:30' to rotate 30 degrees , plot_vrange = or shift:(300,200) to shift by (nrows,ncols). quantities = # range (format vmin:vmax) of values for plotting# (pixel intensity) show_projections = # bool, include projections onto x,y axes, or not. outputfile = # base name for output file. n_hdf5 quantities to plot or save, e.g. 'image dark average maximum' plot_vrange = # range (format vmin:vmax) of values for plotting (pixel intensity) show_projections = # bool, include projections onto x,y axes, or not. outputfile = # temporarily unsupported. base name for output file. max_save = 0 # if larger than zero, will save event images in separate files, up to maximum. calib_path = None # path to calib directory (Currently only needed for CSPad) |
...
pyana -n 10 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_image -m XtcExplorer.pyana_plotter <xtc files>
pyana_ipimb
This module extracts and displays data from IPIMB type detectors. There are , monitoring boards with four diodes each.
The datastream contains two types of information: IpimbData (raw data) and IpmFex (feature extracted data). Only the last type of data is currently implemented, but other data can also be added. Currently the display will show a plot of the intensity measured in each of the four channels, the beam spot position estimated from sums and differences of the individual intensity measurements, and the sum of the four measurements as a function of time. The plot included below has these plots for 4 different IPIMB detector sources.
...
sources # list of IPIMB addresses
quantities # keywords defining what to plot. The following keywords are defined:
# fex:pos plot position based on sum and difference of the 4 diodes
# fex:sum plot sum of signal from all 4 diodes
# fex:channels plot all 4 channels (fex value)
# fex:chX plot fex value from channel X, X=0,1,2,3
# raw:chX plot raw value from channel X, X=0,1,2,3
# raw:chXvolt plot raw voltage from channel X, X=0,1,2,3
What exactly to display is defined in pyana_ipimb.py, but the plotting is done by the DataDisplay object in the displaytools.py library.
This is currently being revised, and has some shortcomings that will be fixed shortly. Currently only one or two diode sources can be displayed at a given time.
Code Block | ||||
---|---|---|---|---|
| ||||
sources # list of IPIMB addresses
quantities # keywords defining what gets stored. Currently the module stores "everything", so this can be ignored.
|
Example of how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_ipimb -m XtcExplorer.pyana_plotter <xtc files>
pyana_epics
Code Block | ||||
---|---|---|---|---|
| ||||
pv # Name(s) of the EPICS PV(s) to dump
|
...
pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_epics -m XtcExplorer.pyana_plotter <xtc files>
pyana_scan
This module is different from the others. In runs with several calibration cycles and one (or more) control PV (process variable), this module displays scalar values from other detectors as a function of scan step of the control PV. It currently takes two scalar type inputs to evaluate the scan:
Code Block | ||||
---|---|---|---|---|
| ||||
input_epics # Name(s) of other scalars to correlate in scan
input_scalars # Name(s) of other scalars to correlate in scan
|
In addition, the control PV name is added to the parameter list. Currently it is only for display, as the module picks up the active control PV withouth using this parameter.
Code Block | ||||
---|---|---|---|---|
| ||||
controlpv = None
|
Note! The XtcExplorer behaviour is different when you check off the ControlPV checkbox! In this case only the pyana_scan module is activated, and all other selected detectors are added as input to this module.
...
pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_scan -m XtcExplorer.pyana_plotter <xtc files>
Helper functions in utilities.py
'utilities.py' is a python module which currently contains several different classes, some related to pyana and others more general, but all developed to help process and plot data with the explorer. This module may soon be split up into several different ones.
Currently it holds the following classes and functions (just a list for now, will add more explanations):
To facilitate parsing and converting the options of pyana:
Code Block none none class PyanaOptions( object ) def __init__( self ): def getOptString(self, options_string) : def getOptStrings(self, options_string) : def getOptStringsDict(self, options_string) : def getOptIntegers(self, options_string): def getOptInteger(self, options_string): def getOptBoolean(self, options_string): def getOptBooleans(self, options_string): def getOptFloats(self, options_string): def getOptFloat(self, options_string):
Container class to store and plot event data
Code Block none none class BaseData( object ): def __init__(self, name,type="BaseData"): def show( self ): def get_plottables_base(self): def get_plottables(self): class BldData( BaseData ): def __init__(self, name, type="BldData"): class IpimbData( BaseData ): def __init__( self, name, type="IpimbData" ): class EncoderData( BaseData ): def __init__( self, name, type="EncoderData" ): class WaveformData( BaseData ): def __init__( self, name, type="WaveformData" ): def get_plottables(self): class EpicsData( BaseData ): def __init__( self, name, type="EpicsData" ): class ScanData( BaseData ) : def __init__(self, name, type="ScanData"): class ImageData( BaseData ): def __init__(self, name, type="ImageData"): def get_plottables(self): class CsPadData( BaseData ): def __init__(self, name, type="CsPadData"):
Container class to keep track of thresholds. Can be passed between modules and from event to event.
Code Block none none class Threshold( object ) :
Container classes to keep track of matplotlib frames and canvases. This can probably be done with
matplotlib's own classes, but I couldn't find out to do it easily. For now, I use these ones:Code Block none none class Frame(object): def __init__(self, name="", title=""): def myticks(self, x, pos): def show(self): def update_axes(self): #def set_ticks_new(self, nticks, lotick, hitick, axis="X"): def set_ticks(self, limits = None ): class Plotter(object): def __init__(self): def add_frame(self, name="", title="",contents=None, type=None, aspect='auto'): def plot_all_frames(self, fignum=1, ordered=False): def settings(self , width = 8 # width of a single plot , height = 7 # height of a single plot , nplots=1 # total number of plots in the figure , maxcol=3 # maximum number of columns ): def create_figure(self, fignum, nplots=1): def close_figure(self): def connect(self,plot=None): def onpick(self, event): def onclick(self, event) : def plot_image(self, image, fignum=1, title="", showProj = False, extent=None): def plot_several(self, list_of_arrays, fignum=1, title="" ): def draw_figurelist(self, fignum, event_display_images, title="",showProj=False,extent=None ) : def draw_figure( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None): def drawframe( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None):
Anchor | ||||
---|---|---|---|---|
|
Interactive plotting with IPython
Since XtcExplorer V00-01-04, you also have the option of launching ipython at the end of the job. The basic support for this has been added to all modules. Each pyana module stores some data (primarily as numpy arrays) which it passes to the pyana_plotter module which launches iPython. In the session you can investigate the data further and make additional plots.
...
A couple of close-ups of the ipython terminal:
xtcscanner
This tool also belongs to the XtcExplorer package, and is used by the GUI. But the tool can also be run directly from the command line:
Code Block |
---|
usage: xtcscanner [options] xtc-files ...
options:
-h, --help show this help message and exit
-n NDATAGRAMS, --ndatagrams=NDATAGRAMS
-v, --verbose
-l L1_OFFSET, --l1-offset=L1_OFFSET
|
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
Scanning....
Start parsing files:
['/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s00-c00.xtc', '/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s01-c00.xtc']
201 datagrams read in 0.070000 s . . . . . . .
-------------------------------------------------------------
XtcScanner information:
- 1 calibration cycles.
- Events per calib cycle:
[197]
Information from 0 control channels found:
Information from 9 devices found
BldInfo:EBeam: EBeamBld (197)
BldInfo:FEEGasDetEnergy: FEEGasDetEnergy (197)
DetInfo:AmoETof-0|Acqiris-0: (5 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197)
DetInfo:AmoGasdet-0|Acqiris-0: (2 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197)
DetInfo:AmoITof-0|Acqiris-0: (1 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197)
DetInfo:AmoMbes-0|Acqiris-0: (1 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197)
DetInfo:EpicsArch-0|NoDevice-0: Epics_V1 (688)
DetInfo:NoDetector-0|Evr-0: EvrConfig_V2 (1)
ProcInfo:: RunControlConfig_V1 (11)
XtcScanner is done!
-------------------------------------------------------------
|
Further analysis with pyana
Any serious data analysis will need more customized tools than we can provide in this general GUI interface.
This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python.
"Blank" analysis code can be generated with Andy's codegen
script (try codegen -h
and codegen -p
for options). Extensive information about pyana is available on confluence.
Running this package's pyana modules outside of the XtcExplorer
Feel free to use any or all of the XtcExplorer modules as examples or starting point for your analysis. If you have suggestions for features or algorithms that might benefit others too, let me know and perhaps we can add it to the XtcExplorer package.
...
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module <xtc files>
Submitting to batch
Once you've identified the command line to run your pyana job, you can submit this to the LFS batch queue system as described in Computing (including Analysis).
Be aware that the pyana scripts used by the explorer use matplotlib QtAgg backend to display plots, and this will fail if you submit it to batch processing. While I look for a better solution, you can do the following to work around this problem:
- Configure pyana to plot as little as possible by setting the display mode to NoDisplay
- Add the following two lines to XtcExplorer/src/utilities.py :
import matplotlib
matplotlib.use('PDF')
A few words on time and performance
The explorer hasn't been optimized for speed properly, but I strive to keep the pyana examples relatively efficient. But keep in mind
that plotting with matplotlib does take a substansial amount of time. An example:
...