The latest tag is V00-01-31.
This is a description and documentation of a GUI tool to explore the XTC data (see Analysis Workbook. Data Formats). The executable is named xtcexplorer, and the software package is named XtcExplorer. The GUI is written in Python/PyQt, while the data is extracted using the LCLS framework pyana (we're currently in process of switching to using psana for this). Visualization of data is obtaned with the plotting package matplotlib, while much of the data algorithms in python relies on NumPy / SciPy libraries.
XTC files contain the raw data (datagrams) streamed from the DAQ online system. The datagrams occur in the order they were flushed from the DAQ system, and thus do not necessarily appear in chronological order. The datagrams are not indexed, so the file must be read sequentially. The pyana and psana frameworks are designed to process these datagrams and present them to you shot by shot (or event by event, in pyana/psana terminology).
For a similar tool to explore the HDF5 data format, look at HDF5 Explorer - Old.
[user@pslogin ~] newrel ana-current myrelease [user@pslogin ~] cd myrelease [user@pslogin myrelease] sit_setup |
XtcExplorer
package from the SVN software repository:
pslogin
is automatic (I think), but if you're on psananeh/psanafeh, issue a kinit
command first:
[user@pslogin myrelease] kinit Password for user@SLAC.STANFORD.EDU: |
[user@pslogin myrelease] addpkg XtcExplorer [user@pslogin myrelease] scons |
addpkg XtcExplorer VXX-XX-XX
' to get a specific version of the package, or 'addpkg XtcExplorer HEAD
' to get the very latest version.
psananeh/psanafeh
for this:
[user@psanaXXX myrelease] sit_setup [user@psanaXXX myrelease] xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0* |
|
The following documentation describes XtcExplorer V00-01-31. For earlier versions, see earlier versions of this confluence page (go to "Tools" -> "Page history").
Typing xtcexplorer will launch the main GUI, which looks something like this:
You need to connect an XTC file in order to get any further. You have 4 options:
xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0*
Once a file or more has been connected, another GUI will pop upĀ (possibly after a short delay while the file(s) content is being investigated). This GUI displays information about the file content and help with further processing of the data in pyana.
The new GUI (Pyana Control Center) will show you a list of detectors/devices found in the file. A little more information is written to the terminal window too.
|
The Pyana Control Center. |
|
Some more information is printed to the terminal window from which the xtcexplorer was launched |
|
Once you checkmark the detectors you want to display information from, another tab will pop up showing pyana configuration text. "Current pyana configuration": as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field. |
More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).
The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!
XtcExplorer/src/pyana_bld.py # display of Beam-line data XtcExplorer/src/pyana_epics.py # display of Epics PV data XtcExplorer/src/pyana_scan.py # display of motor scan data XtcExplorer/src/pyana_encoder.py # display of encoder data XtcExplorer/src/pyana_ipimb.py # display of diode data from IPIMB and PIM XtcExplorer/src/pyana_waveform.py # display of waveform data XtcExplorer/src/pyana_image.py # display of camera image data XtcExplorer/src/pyana_plotter.py # a plotter module to control the event display |
Additional helper classes and functions are defined in
XtcExplorer/src/utilities.py # PyanaOptions - help parse options # BaseData and classes derived thereof - store numpy arrays for ipython # Threshold - keep track of threshold values and region of interest # Frame - container for a single "plot" # Plotter - container for a single "canvas" XtcExplorer/src/cspad.py # CsPad - for arranging & aligning the CsPad elements |
These are imported into the pyana modules wherever needed. You might want to keep this in mind when
using examples from this package. The pyana modules can mostly be run on their own, but they might
need to import this module.
A few things to note about the different detectors / pyana modules:
A few parameters are common to all modules, and their default values are set to be the same, but can also be set individually for each module. They are the following:
# parameter name default value comment plot_every_n = 100 # If 0, plot only at the end. Else, display every N events accumulate_n = 0 # If 0, accumulate all (no reset). Else, reset stored arrays every N events fignum = 1 # matplotlib figure number |
This module is added to the end of the job, after other modules. It controls the display mode and ipython.
display_mode = 2 # Interactive (1) or SlideShow (2) or NoDisplay (0) ipython = False # Drop into ipython after each plot (to allow further investigation of data) |
This module collects data from the BeamLine Detectors (Bld) and plots them in separate windows.
do_ebeam = False # Plot data from EBeam object do_gasdetector = False # Plot data from GasDetector do_phasecavity = False # Plot data from PhaseCavity do_ipimb = False # Plot data from SharedIpimb (No longer used, see pyana_ipimb) |
Example of how to run this pyana module by itself, i.e. no GUI, processing 300 shots and using an xtcexplorer-generated config file:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_bld -m XtcExplorer.pyana_plotter <xtc files>
Plots waveform data from Acqiris digitizers. Takes a list of addresses, currently will make one plot for each channel of each device. Plan to add more options.
Configurable options:
sources = # string, space-separated list of addresses (Detector-Id|Device-ID) quantities = # string containing quantities to plot. "average stack single" |
Example how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform -m XtcExplorer.pyana_ploter <xtc files>
Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform <xtc files>
This module deals with all camera images, including CsPad images.
Options must be specified in a configuration file. Only the sources parameter is required.
sources = # string, Address of Detector-Id|Device-ID inputdark = # filename of a binary numpy (.npy) with dark image for subtraction threshold = # Threshold information, specify the keywords that apply: lower=value, upper=value, type=maximum (default) or average, roi=(xmin:xmax,ymin:ymax)' algorithms = # algorithms to be applied to the image, e.g.: 'rotate:30' to rotate 30 degrees, or shift:(300,200) to shift by (nrows,ncols). quantities = # quantities to plot or save, e.g. 'image dark average maximum' plot_vrange = # range (format vmin:vmax) of values for plotting (pixel intensity) show_projections = # bool, include projections onto x,y axes, or not. outputfile = # base name for output file. max_save = 0 # if larger than zero, will save event images in separate files, up to maximum. calib_path = None # path to calib directory (Currently only needed for CSPad) |
|
CsPad image where the region of interest (a red rectangle in the left figure) has intensity above a given threshold. The right image shows the region of interest. |
|
CsPad image where region of interest (red rectangle) has intensity above a given threshold. Additionally, a dark-subtracted image is shown (bottom left), where the dark average is based on below-threshold shots from the same run (shown bottom right. |
|
This image is similar to the one above. The intensity limits have been adjusted interactively. |
|
At the end of the job, the average of good images (left) and rejected / dark images (right) is shown. |
|
Image from two Pulinx cameras, no filtering or background subtraction. Projections in x and y are shown. |
|
Displays of three different Pulnix TM6740 images of YAG screens, after rotation/translation. Also shown, differences between images and FFT of differences. |
|
Display of images from Princeton camera. Plotted at end of job of average images above threshold, and average images below threshold (dark). A red rectangle shows the region of interest that was used to evaluate intensity w.r.t. threshold. Configuration: 'threshold = 350 (700:800,1100:1200)' |
Example of how to run this pyana module by itself, i.e. no GUI, processing 10 events and using an xtcexplorer-generated config file:
pyana -n 10 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_image -m XtcExplorer.pyana_plotter <xtc files>
This module extracts and displays data from IPIMB type detectors, monitoring boards with four diodes each.
The datastream contains two types of information: IpimbData (raw data) and IpmFex (feature extracted data).
What exactly to display is defined in pyana_ipimb.py, but the plotting is done by the DataDisplay object in the displaytools.py library.
This is currently being revised, and has some shortcomings that will be fixed shortly. Currently only one or two diode sources can be displayed at a given time.
sources # list of IPIMB addresses quantities # keywords defining what gets stored. Currently the module stores "everything", so this can be ignored. |
Example of how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_ipimb -m XtcExplorer.pyana_plotter <xtc files>
pv # Name(s) of the EPICS PV(s) to dump |
Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:
pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_epics -m XtcExplorer.pyana_plotter <xtc files>
This module is different from the others. In runs with several calibration cycles and one (or more) control PV (process variable), this module displays scalar values from other detectors as a function of scan step of the control PV. It currently takes two scalar type inputs to evaluate the scan:
input_epics # Name(s) of other scalars to correlate in scan input_scalars # Name(s) of other scalars to correlate in scan |
In addition, the control PV name is added to the parameter list. Currently it is only for display, as the module picks up the active control PV withouth using this parameter.
controlpv = None |
Note! The XtcExplorer behaviour is different when you check off the ControlPV checkbox! In this case only the pyana_scan module is activated, and all other selected detectors are added as input to this module.
Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:
pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_scan -m XtcExplorer.pyana_plotter <xtc files>
'utilities.py' is a python module which currently contains several different classes, some related to pyana and others more general, but all developed to help process and plot data with the explorer. This module may soon be split up into several different ones.
Currently it holds the following classes and functions (just a list for now, will add more explanations):
class PyanaOptions( object ) def __init__( self ): def getOptString(self, options_string) : def getOptStrings(self, options_string) : def getOptStringsDict(self, options_string) : def getOptIntegers(self, options_string): def getOptInteger(self, options_string): def getOptBoolean(self, options_string): def getOptBooleans(self, options_string): def getOptFloats(self, options_string): def getOptFloat(self, options_string): |
class BaseData( object ): def __init__(self, name,type="BaseData"): def show( self ): def get_plottables_base(self): def get_plottables(self): class BldData( BaseData ): def __init__(self, name, type="BldData"): class IpimbData( BaseData ): def __init__( self, name, type="IpimbData" ): class EncoderData( BaseData ): def __init__( self, name, type="EncoderData" ): class WaveformData( BaseData ): def __init__( self, name, type="WaveformData" ): def get_plottables(self): class EpicsData( BaseData ): def __init__( self, name, type="EpicsData" ): class ScanData( BaseData ) : def __init__(self, name, type="ScanData"): class ImageData( BaseData ): def __init__(self, name, type="ImageData"): def get_plottables(self): class CsPadData( BaseData ): def __init__(self, name, type="CsPadData"): |
class Threshold( object ) : |
class Frame(object): def __init__(self, name="", title=""): def myticks(self, x, pos): def show(self): def update_axes(self): #def set_ticks_new(self, nticks, lotick, hitick, axis="X"): def set_ticks(self, limits = None ): class Plotter(object): def __init__(self): def add_frame(self, name="", title="",contents=None, type=None, aspect='auto'): def plot_all_frames(self, fignum=1, ordered=False): def settings(self , width = 8 # width of a single plot , height = 7 # height of a single plot , nplots=1 # total number of plots in the figure , maxcol=3 # maximum number of columns ): def create_figure(self, fignum, nplots=1): def close_figure(self): def connect(self,plot=None): def onpick(self, event): def onclick(self, event) : def plot_image(self, image, fignum=1, title="", showProj = False, extent=None): def plot_several(self, list_of_arrays, fignum=1, title="" ): def draw_figurelist(self, fignum, event_display_images, title="",showProj=False,extent=None ) : def draw_figure( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None): def drawframe( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None): |
Since XtcExplorer V00-01-04, you also have the option of launching ipython at the end of the job. The basic support for this has been added to all modules. Each pyana module stores some data (primarily as numpy arrays) which it passes to the pyana_plotter module which launches iPython. In the session you can investigate the data further and make additional plots.
Here are a few screenshot examples:
Full screenshot of all GUIs, plots and ipython session:
A couple of close-ups of the ipython terminal:
This tool also belongs to the XtcExplorer package, and is used by the GUI. But the tool can also be run directly from the command line:
usage: xtcscanner [options] xtc-files ... options: -h, --help show this help message and exit -n NDATAGRAMS, --ndatagrams=NDATAGRAMS -v, --verbose -l L1_OFFSET, --l1-offset=L1_OFFSET |
Example:
Scanning.... Start parsing files: ['/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s00-c00.xtc', '/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s01-c00.xtc'] 201 datagrams read in 0.070000 s . . . . . . . ------------------------------------------------------------- XtcScanner information: - 1 calibration cycles. - Events per calib cycle: [197] Information from 0 control channels found: Information from 9 devices found BldInfo:EBeam: EBeamBld (197) BldInfo:FEEGasDetEnergy: FEEGasDetEnergy (197) DetInfo:AmoETof-0|Acqiris-0: (5 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197) DetInfo:AmoGasdet-0|Acqiris-0: (2 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197) DetInfo:AmoITof-0|Acqiris-0: (1 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197) DetInfo:AmoMbes-0|Acqiris-0: (1 ch) AcqConfig_V1 (1) AcqWaveform_V1 (197) DetInfo:EpicsArch-0|NoDevice-0: Epics_V1 (688) DetInfo:NoDetector-0|Evr-0: EvrConfig_V2 (1) ProcInfo:: RunControlConfig_V1 (11) XtcScanner is done! ------------------------------------------------------------- |
pyana
Any serious data analysis will need more customized tools than we can provide in this general GUI interface.
This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python.
"Blank" analysis code can be generated with Andy's codegen
script (try codegen -h
and codegen -p
for options). Extensive information about pyana is available on confluence.
Feel free to use any or all of the XtcExplorer modules as examples or starting point for your analysis. If you have suggestions for features or algorithms that might benefit others too, let me know and perhaps we can add it to the XtcExplorer package.
Example how to run a pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module -m XtcExplorer.pyana_ploter <xtc files>
Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots, you can also do without the pyana_plotter module:
pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module <xtc files>
Once you've identified the command line to run your pyana job, you can submit this to the LFS batch queue system as described in Computing (including Analysis).
Be aware that the pyana scripts used by the explorer use matplotlib QtAgg backend to display plots, and this will fail if you submit it to batch processing. While I look for a better solution, you can do the following to work around this problem:
import matplotlib
matplotlib.use('PDF')
The explorer hasn't been optimized for speed properly, but I strive to keep the pyana examples relatively efficient. But keep in mind
that plotting with matplotlib does take a substansial amount of time. An example:
Running the explorer on a small xtc file from a XCS commissioning run, containing 1267 events, and plotting information
from two IPIMB detectors and a YAG camera image.
So matplotlib drawing these plots take about 1.5 s each time, quite substantial compared to 5.7 microseconds for pyana to process each event. Other types of plots may take longer to plot, e.g. larger images, waveforms, etc.