Table of Contents

maxLevel	3

xtcexplorer (GUI interface to pyana)

Note! This package is a new version of the "XtcEventBrowser" which is the old name of the packageThe latest tag is V00-01-31.

This is a description and documentation of a GUI tool to explore the xtc XTC data . It (see Analysis Workbook. Data Formats). The executable is named xtcexplorer, and the software package is named XtcExplorer. The GUI is written in python, relying on PyQt4 for graphical user interface. The data processing is done via the pyana framework and visualization provided by matplotlib. It can be run on any xtc file, including the file being written during data aquisition (DAQ). Python/PyQt, while the data is extracted using the LCLS framework pyana (we're currently in process of switching to using psana for this). Visualization of data is obtaned with the plotting package matplotlib, while much of the data algorithms in python relies on NumPy / SciPy libraries.

XTC Xtc files contain the raw data (datagrams) streamed from the DAQ online system. The datagrams occur in the order they were flushed from the DAQ system, and thus do not necessarily appear in chronological order. The datagrams are not indexed, so the file must be read sequentially. The pyana and psana frameworks are designed to process these datagrams and present them to you shot by shot (or event by event, in pyana/psana terminology).

The XtcExplorer allows a simple-to-run interface to the xtc files. The package name is XtcExplorer, the executables (xtcexplorer and xtcscanner) are found in the app subdirectory of this package, and all other source code is in the src subdirectory.

For a similar tool to explore the HDF5 data format, look at HDF5 ExplorerFor a similar tool to explore the HDF5 data format, look at HDF5 Explorer - Old.

How to get started

Set up your release work directory:
You should always run this code from your own release working directory.

...

See Account Setupfor more details on how to set up the analysis environment

...

:
Code Block
none
none

...

[user@pslogin ~] newrel ana-current myrelease
[user@pslogin ~] cd myrelease
[user@pslogin myrelease] sit_setup

...

Check out the XtcExplorer package

...

from the SVN software repository:
This step is optional. But without it, you will not be able to edit/customize any source files.
You need an afs token to check out the package

...

, which on pslogin is automatic (I think), but if you're on psananeh/psanafeh, issue a kinitcommand first:
Code Block
none
none

...

[user@pslogin myrelease] kinit
Password for user@SLAC.STANFORD.EDU:

...

Now you're ready to check out the package and "compile":
Code Block
none
none
[user@pslogin myrelease] addpkg XtcExplorer

...

[user@pslogin myrelease] scons
Note!

...

You can also do 'addpkg XtcExplorer VXX-XX-XX

...

' to get a specific version of the package, or 'addpkg XtcExplorer HEAD' to get the very latest

...

version

...

.
Run the program:
Launch the GUI with the command 'xtcexplorer' and

...

optionaly give the input xtc files that you want to read
as arguments. You can also browse to find files after launching the browser.

To have access to the xtc files, you need to login to

...

psananeh/psanafehfor this:
Code Block

...

[user@psanaXXX myrelease] sit_setup
[user@psanaXXX myrelease] xtcexplorer /reg/d/psdm/

...

CXI/

...

cxi22010/xtc/

...

e60-r0465-

...

s0*

Info
If you encounter any problems, ask for help! Don't be stuck. This is supposed to be easy. If it's not, I did something wrong, so let me know! Send

...

us an

...

e-mail (pcds-help at slac.stanford.edu)

Description of the GUIs

LCLS Xtc Explorer

xtcexplorer will launch the main GUI. It currently has two sections:

File section: Shows a list of currently selected file(s). As you may have guessed, "File Browser" opens a file browser and "Clear File List" clears the current list of files. This section also allows you to add a file name by hand (or paste).
Scan section: The two buttons to the left allows you to scan the xtc file to get a summary of what datagrams are stored in it. Note, for most purposes, a "Quick Scan" is sufficient. If you need to scan the whole file, e.g. if you want to know the total number of events, number of calibration cycles, etc, you can enable the "Scan File(s)" button. If the files are big, this will take a lot of time...

Image Removed	The main window before any file selection. Click on "File Browser..." to select file(s).
Image Removed	A standard file browser pops up. Navigate the file browser till you find the xtc files for your experiment. (Remember, if this seems cumbersome, you can also give the file name(s) as argument to xtcexplorer when you first launch the GUI)
Image Removed	Mark the files you want to look at (Hold down the Ctrl-key to select more than one file). Note, you should only look at one run number at a time, since different runs might have different configurations.
Image Removed	This is the main window after a file has been selected. File name and file size is shown in the GUI. If the file is not too big, you can click the "Scan File(s)" button to get exact contents of the whole file. If the file is big, it's better to do a "Quick Scan" which will tell you all you need to know (except count number of events and calibration cycles).

Pyana Control Center

xtcexplorer writes out pyana configuration files, based on selections made in the GUI. Look out for files with names xb_pyana_XXXX.cfg, where XXXX is a random four-digit number. You can use them to run pyana directly from the command line. But they accumulate, so you might want to remove these regularly...
LCLS data is very diverse, and not all features for all detectors are implemented (yet). Feel free to contact me if you have requests and suggestions.

Description of the GUIs

The following documentation describes XtcExplorer V00-01-31. For earlier versions, see earlier versions of this confluence page (go to "Tools" -> "Page history").

LCLS XTC Explorer

Typing xtcexplorer will launch the main GUI, which looks something like this:

Image Added

You need to connect an XTC file in order to get any further. You have 4 options:

You can give file names as arguments on the command line, and the file will automatically be opened/connected< e.g. xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0*
If you know the Instrument / Experiment number / run number you'd like to investigate, simply select from the pull-down menus.
There's also a 'File Browser' that allows you to browse for the files you want. Be aware that many runs are split into several files, in chunks and streams, and to get a complete dataset you do need to use all of the files for a given run. It will probably 'work' for single files too, but we cannot guarantee completely synchronized date, in particular, EPICS data is stored at a different rate than 'event' (shot) data, so if you make use of EPICS PV data, use the complete dataset.
Finally, you have the option of adding file names by hand.

Once a file or more has been connected, another GUI will pop up (possibly after a short delay while the file(s) content is being investigated). This GUI displays information about the file content and help with further processing of the data in pyana.

Image Added

Pyana Control Center

The new GUI (Pyana Control Center) will show After scanning, a new GUI will pop up showing you a list of detectors/devices found in the file. A little more information is written to the terminal window too.

Image Added	The	Image RemovedImage Removed	Main window (left) and Pyana Control Center (right) after the file scan. The main window gives you a short summary of how many events and how many calibration cycles were found. Some more information is printed to the terminal window from which the xtcbrowser was launched The pyana control center has a field to the left that shows you the detector/device sources found within the file(s). In front of each detector/device name is a checkbox, where you can select which datagrams you are interested in analysing / plotting. To the right of this is a field with information and general settings: How many events to process total, skip events at the beginning of the file(s), How often to display plot (default is every 100 events, unless several scan steps, in which case it will plot for each new scan step).. In the pyana control center, select general run options and plot options from the panel to the right, and detectors/devices from the list to the left. Once you checkmark the detectors, a new "Pyana Configuration" tab pops up over the run/plot option panel. You can always go back to the "General Settings" tab. "General Settings" have two components: run options (number of events to run, to skip, and multiprocessing), and plot options (frequency of plotting, display mode, ipython at end job). A note on multiprocessing: You are allowed to select how many CPUs to use, mostly to advertise this possibility when running pyana. It is not possible to interact with all subprocesses, so the XtcExplorer will only be able to make plots from one of the subprocesses.
Image Added	Some more information is printed to the terminal window from which the xtcexplorer was launched
Image Added Image Added	Once you checkmark the detectors you want to display information from, another tab will	Image Removed Image Removed	Once you checkmark the detectors you want to display information from, another tab will pop up showing pyana configuration text. "Current pyana configuration": as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field. If a ControlPV is present and checked off, only a pyana_scan module will be used. All the other devices you check will be added to the input of the scan. If no ControlPV is used, other pyana modules will be configured as appropriate to display a variety of information from the events. If "Epics Process Variables" are checked off, another Gui appears that lists all the epics variables. Select the ones you want to display. Press the "Write configuration to file" button once you're done. You can further edit the file by hand if you want. Once a file is written, a "Run pyana" button will appear. You can further edit the configuration file by hand. If you click on "Edit configuration", the file will open in the default editor. Note! The default editor is defined in your EDITOR environment variable. Make sure it is set to your editor of choice (e.g. `setenv EDITOR emacs`). "Run pyana" lauches an input GUI that shows you the runstring. You can use the same runstring from the command line. Or hit "OK" and it'll run. After launching pyana, another button "Quit pyana" appears... If you see you need to change parameters, you can stop pyana, edit the configuration file, and start over again.

More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).

The pyana modules

The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!

Code Block

	none
	none


XtcExplorer/src/pyana_bld.py          # display of Beam-line data
XtcExplorer/src/pyana_cspadepics.py        # display of CsPadEpics imagePV data
XtcExplorer/src/pyana_imagescan.py         # display of cameramotor imagescan data
XtcExplorer/src/pyana_ipimbencoder.py        # display of diodeencoder data from IPIMB and PIM
XtcExplorer/src/pyana_waveformipimb.py        # display of waveformdiode data from IPIMB and PIM
XtcExplorer/src/pyana_epicswaveform.py        # display of Epics PVwaveform data
XtcExplorer/src/pyana_scanimage.py         # display of motorcamera scanimage data
XtcExplorer/src/pyana_plotter.py      # a plotter module to control the event display

A few things to note about the different detectors / pyana modules:

Common configuration parameters

A few parameters are common to all modules, and their default values are set to be the same, but can also be set individually for each module. They are the following:

Additional helper classes and functions are defined in

Code Block

	none
	none

XtcExplorer/src/utilities.py          # PyanaOptions

Code Block

none


# parameter name   default value   comment
  plot_every_n      = 100         # If 0, plot only at the end. Else, display every N events
  fignum- help parse options
         = 1           # matplotlib figure number

pyana_plotter

This module is added to the end of the job, after other modules. It does nothing (yet) than control the display mode. Default is SlideShow mode. There's also a handle to allow you to use ipython at the end of the job (or after each plot) to investigate your plottet arrays further. This is not functional yet.

Code Block

none


  display_mode                 # BaseData and classes derived thereof      - store numpy arrays for ipython
                          = 2           # Threshold Interactive (1) or SlideShow (2) or NoDisplay (0)
  ipython           = False       #  Drop into ipython- afterkeep eachtrack plotof (tothreshold allowvalues furtherand investigationregion of data)

pyana_bld

This module collects data from the BeamLine Detectors (Bld) and plots them in separate windows. In SlideMode, the plots accumulate from one plot to the next.
To run this module by itself with pyana:

pyana -m XtcExplorer/src/pyana_waveform.py <xtc files>

Code Block

none


 do_ebeaminterest
                                      # =Frame False       #  Plot data from EBeam object
 do_gasdetector     = False       #  Plot data from GasDetector
 do_phasecavity   - container =for Falsea single "plot"
                                      # Plotter  Plot data from PhaseCavity
 do_ipimb                         = False       #- container Plotfor dataa from SharedIpimb

Image Removed Image Removed Image Removed

pyana_waveform

Plots waveform data from Acqiris digitizers. Takes a list of addresses, currently will make one plot for each channel of each device.
To run this module by itself with pyana:

pyana -m XtcExplorer/src/pyana_waveform.py <xtc files>

Configurable options:

Code Block
none	none	sourcessingle "canvas" XtcExplorer/src/cspad.py # =CsPad # string, space-separated list of addresses (Detector-Id\|Device-ID)

Image Removed

pyana_cspad

CsPad data is reconstructed in pyana_cspad.py. The image plot value limits are adjusted automatically, but if
you want to change them, click on the color bar (left-click for low limit, right-click for high limit).
The successive events will be plotted with the new limits. Revert to the original by middle-clicking.
To run this module by itself with pyana:

pyana -m XtcExplorer/src/pyana_cspad.py <xtc files>

                   - for arranging & aligning the CsPad elements

These are imported into the pyana modules wherever needed. You might want to keep this in mind when
using examples from this package. The pyana modules can mostly be run on their own, but they might
need to import this module.

...

A few things to note about the different detectors / pyana modules:

Common configuration parameters

A few parameters are common to all modules, and their default values are set to be the same, but can also be set individually for each module. They are the following:Options must be specified in a configuration file. Only the img_sources parameter is required.

Code Block

	none
	none


# parameter img_sourcesname   default value   =comment
  plot_every_n      = 100    # string, Address of Detector-Id|Device-ID
  dark_img_file     =         # If 0, plot only at the end. Else, display every N events
  accumulate_n     # filename,= Dark0 image file to be loaded, if any
  out_img_file  # If 0, accumulate =all (no reset). Else, reset stored arrays every N events
  fignum  # filename (If collecting: write the average image to this file)
  plot_vrange       = 1            # range=vmin,vmax of values for plotting (pixel intensity)
  threshold         =             # threshold for image intensity in active threshold area
  thr_area          =             # active threshold area (range=xmin,xmax,ymin,ymax). None == whole image.

Image Removed

Image display of the CSPad detector, background subtracted. This is currently the only display that has interactive features. A filter allows you to select events within requested intensity range.

pyana_image.py

This module processes generic camera frames, e.g from Pulnix TM6740 device. It allows any number of images, given as a space-separated list of addresses in the configuration file.

matplotlib figure number

pyana_plotter

This module is added to the end of the job, after other modules. It controls the display mode and ipython.

The default display mode is SlideShow mode, where the plot window stays open but updates every n events. In Interactive mode, the figure window must be closed before the next event gets processed. The other modules are set up to work with this one, but should with minor modifications also run fine without it. (The minor modifications would be to issue a "plt.ion()" before the figures are created or replace "plt.draw()" with "plt.show()" to make sure the plot windows stay open.)
There's also a handle to allow you to use ipython at the end of the job (or after each plot) to investigate your plottet arrays further.
Code Block
none
none
display_mode = 2 # Interactive (1) or SlideShow (2) or NoDisplay (0) ipython = False
You can set ranges to define good images and dark images. If both are set, you have the option to display good images background subtracted, where background subtraction is based on the average of background images so far collected.
Each image can be separately rotated, shifted and scaled (zoomed in/out).
Nicknames can be given to the input images. Defaults are Im1, Im2... etc. These names will be used if you plot differences, or other manipulations of the original images.

The images are subtracted and differences displayed as well as fourier transform of differences. Examples of what may be displayed. To display other things, at this stage you have to edit pyana_image.py to change this behaviour.

Code Block

none


 image_addresses     # address string of Detector-Id|Device-ID
 good_range          # threshold Drop valuesinto selectingipython imagesafter ofeach interestplot (Format: low,high)
 dark_rangeto allow further investigation of data)

pyana_bld

This module collects data from the BeamLine Detectors (Bld) and plots them in separate windows.

Code Block

	none
	none

 do_ebeam

...

False

...

 #  Plot data

...

from

...

EBeam

...

object
 do_gasdetector     = False       #

...

Plot

...

data

...

from

...

GasDetector
 do_phasecavity     = False

...

Plot

...

data

...

from PhaseCavity

...

do_

...

ipimb

...

False

...

Plot

...

data

...

from

...

SharedIpimb

...

No

...

longer

...

used, see pyana_ipimb)

Image Added Image Added Image Added Image Added

Example of how to run this pyana module by itself, i.e. no GUI, processing 300 shots and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_bld -m XtcExplorer.pyana_plotter <xtc files>

pyana_waveform

Plots waveform data from Acqiris digitizers. Takes a list of addresses, currently will make one plot for each channel of each device. Plan to add more options.

Configurable options:

Code Block

	none
	none

  sources

Image Removed

Displays of three different Pulnix TM6740 images of YAG screens, after rotation/translation. Also shown, differences between images and FFT of differences.

pyana_ipimb

...


ipimb_addresses     # list of IPIMB addresses

Image Removed

pyana_epics

Code Block
none	none	pv = # Name(s) of the EPICS PV(s) to dump

pyana_scan

This module is different from the others. It does a "motor scan", displays certain values as a function of scan step of a motor scan. The motor name is given in the xtc file as the Control PV. There may be more than one control PV. It currently takes two scalar type inputs to evaluate the scan:

Code Block
none	none	input_epics string, space-separated list of addresses (Detector-Id\|Device-ID) quantities = # Name(s) of other scalars to correlate in scan input_scalars # Name(s) of other scalars to correlate in scan

Image Removed

xtcscanner

This tool also belongs to the XtcExplorer package, and is used by the GUI. But the tool can also be run directly from the command line:

Code Block
usage: xtcscanner [options] xtc-files ... options: -h, --help show this help message and exit -n NDATAGRAMS, --ndatagrams=NDATAGRAMS -v, --verbose -l L1_OFFSET, --l1-offset=L1_OFFSET

Example:

# string containing quantities to plot. "average stack single"

Image Added

Example how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform -m XtcExplorer.pyana_ploter <xtc files>

Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform <xtc files>

pyana_image

This module deals with all camera images, including CsPad images.

Load dark image from a file for background subtraction
Filter events: the 'threshold' parameter holds settings for filtering events based on image values. Specify keywords=value separated by whitespaces, e.g.
- 'lower=400.0 upper=1600.0 type=maximum roi=(200:300,100:400)' which will filter out events that have the maximum pixel in the region of interest below 400 or above 1600.
- 'lower=500 type=average' will filter out events that have average pixel value in the whole image below 500.
Apply algorithms to an image (e.g. rotation, shift, etc)
Display and/or save the image, a background subtacted image, an average image, an image with algorithms applied... etc.
Save the average image(s) or selected single-shot images (up to a maximum) to file
In Interactive mode (see pyana_plotter), some interaction with the plot is possible:
- change intensity range (color range) by left-clicking to set low limit, right-clicking to set high limit, middle-click to reset to full scale.
- change threshold value and region of interest if a ROI was defined: Click on the rectangle, enter new values when prompted in terminal window
- switch to SlideShow mode when you are content with the display: Middle-click on the canvas and enter new display mode value (2 for SlideShow).

Options must be specified in a configuration file. Only the sources parameter is required.

Code Block

	none
	none

  sources      =        # string, Address of Detector-Id|Device-ID
  inputdark    =

Code Block

none

title

xtcscanner -n 200 /reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s0*


Scanning....
Start parsing files:
['/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s00-c00.xtc', '/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s01-c00.xtc']
  201 datagrams read in 0.070000 s .   .   .   .   .   .   .
-------------------------------------------------------------
XtcScanner information:
  - 1 calibration cycles.
  - Events per calib cycle:
   [197]

Information from  0  control channels found:
Information from  9  devices found
        # filename of a binary numpy (.npy) with dark image for subtraction
  threshold BldInfo:EBeam:   =        # Threshold EBeamBld (197)
            BldInfo:FEEGasDetEnergy:    information, specify the keywords that apply: lower=value, upper=value, type=maximum (default) or average, roi=(xmin:xmax,ymin:ymax)'
  algorithms   =        # FEEGasDetEnergy (197)
        DetInfo:AmoETof-0|Acqiris-0:  (5 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
      DetInfo:AmoGasdet-0|Acqiris-0:  (2 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoITof-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoMbes-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
     DetInfo:EpicsArch-0|NoDevice-0:             Epics_V1 (688)
         DetInfo:NoDetector-0|Evr-0:             EvrConfig_V2 (1)
                          ProcInfo::             RunControlConfig_V1 (11)
XtcScanner is done!
-------------------------------------------------------------

Further analysis with `pyana`

Any serious data analysis will need more customized tools than we can provide in a GUI interface. This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python. The Gui Event Browser can provide simple analysis code that can be expanded by the user. "Blank" analysis code can also be generated with Andy's codegen script (try codegen -h and codegen -p for options).

algorithms to be applied to the image, e.g.: 'rotate:30' to rotate 30 degrees, or shift:(300,200) to shift by (nrows,ncols).
  quantities   =        # quantities to plot or save, e.g. 'image dark average maximum'
  plot_vrange  =        # range (format vmin:vmax) of values for plotting (pixel intensity)
  show_projections =    # bool, include projections onto x,y axes, or not.
  outputfile   =        # base name for output file.
  max_save     = 0      # if larger than zero, will save event images in separate files, up to maximum.
  calib_path   = None   # path to calib directory (Currently only needed for CSPad)

Image Added	CsPad image where the region of interest (a red rectangle in the left figure) has intensity above a given threshold. The right image shows the region of interest.
Image Added	CsPad image where region of interest (red rectangle) has intensity above a given threshold. Additionally, a dark-subtracted image is shown (bottom left), where the dark average is based on below-threshold shots from the same run (shown bottom right.
Image Added	This image is similar to the one above. The intensity limits have been adjusted interactively.
Image Added	At the end of the job, the average of good images (left) and rejected / dark images (right) is shown.
Image Added	Image from two Pulinx cameras, no filtering or background subtraction. Projections in x and y are shown.
Image Added	Displays of three different Pulnix TM6740 images of YAG screens, after rotation/translation. Also shown, differences between images and FFT of differences.
Image Added	Display of images from Princeton camera. Plotted at end of job of average images above threshold, and average images below threshold (dark). A red rectangle shows the region of interest that was used to evaluate intensity w.r.t. threshold. Configuration: 'threshold = 350 (700:800,1100:1200)'

Example of how to run this pyana module by itself, i.e. no GUI, processing 10 events and using an xtcexplorer-generated config file:

pyana -n 10 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_image -m XtcExplorer.pyana_plotter <xtc files>

pyana_ipimb

This module extracts and displays data from IPIMB type detectors, monitoring boards with four diodes each.
The datastream contains two types of information: IpimbData (raw data) and IpmFex (feature extracted data).
What exactly to display is defined in pyana_ipimb.py, but the plotting is done by the DataDisplay object in the displaytools.py library.
This is currently being revised, and has some shortcomings that will be fixed shortly. Currently only one or two diode sources can be displayed at a given time.

Code Block

	none
	none

sources     # list of IPIMB addresses
quantities  # keywords defining what gets stored. Currently the module stores "everything", so this can be ignored.

Image Added Image Added
Example of how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_ipimb -m XtcExplorer.pyana_plotter <xtc files>

pyana_epics

Code Block

	none
	none

pv                  # Name(s) of the EPICS PV(s) to dump

Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:

pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_epics -m XtcExplorer.pyana_plotter <xtc files>

pyana_scan

This module is different from the others. In runs with several calibration cycles and one (or more) control PV (process variable), this module displays scalar values from other detectors as a function of scan step of the control PV. It currently takes two scalar type inputs to evaluate the scan:

Code Block

	none
	none

input_epics         # Name(s) of other scalars to correlate in scan
input_scalars       # Name(s) of other scalars to correlate in scan

In addition, the control PV name is added to the parameter list. Currently it is only for display, as the module picks up the active control PV withouth using this parameter.

Code Block

	none
	none

controlpv = None

Note! The XtcExplorer behaviour is different when you check off the ControlPV checkbox! In this case only the pyana_scan module is activated, and all other selected detectors are added as input to this module.

Image Added Image Added Image Added

Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:

pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_scan -m XtcExplorer.pyana_plotter <xtc files>

Helper functions in utilities.py

'utilities.py' is a python module which currently contains several different classes, some related to pyana and others more general, but all developed to help process and plot data with the explorer. This module may soon be split up into several different ones.

Currently it holds the following classes and functions (just a list for now, will add more explanations):

To facilitate parsing and converting the options of pyana:

Code Block

	none
	none

  class PyanaOptions( object )
    def __init__( self ):
    def getOptString(self, options_string) :
    def getOptStrings(self, options_string) :
    def getOptStringsDict(self, options_string) :
    def getOptIntegers(self, options_string):
    def getOptInteger(self, options_string):
    def getOptBoolean(self, options_string):
    def getOptBooleans(self, options_string):
    def getOptFloats(self, options_string):
    def getOptFloat(self, options_string):

Container class to store and plot event data

Code Block

	none
	none

  class BaseData( object ):
   def __init__(self, name,type="BaseData"):
    def show( self ):
    def get_plottables_base(self):
    def get_plottables(self):

  class BldData( BaseData ):
    def __init__(self, name, type="BldData"):

  class IpimbData( BaseData ):
    def __init__( self, name, type="IpimbData" ):

  class EncoderData( BaseData ):
    def __init__( self, name, type="EncoderData" ):

  class WaveformData( BaseData ):
    def __init__( self, name, type="WaveformData" ):
    def get_plottables(self):

  class EpicsData( BaseData ):
    def __init__( self, name, type="EpicsData" ):

  class ScanData( BaseData ) :
    def __init__(self, name, type="ScanData"):

  class ImageData( BaseData ):
    def __init__(self, name, type="ImageData"):
    def get_plottables(self):

  class CsPadData( BaseData ):
    def __init__(self, name, type="CsPadData"):

Container class to keep track of thresholds. Can be passed between modules and from event to event.
Code Block
none
none
class Threshold( object ) :

Container classes to keep track of matplotlib frames and canvases. This can probably be done with
matplotlib's own classes, but I couldn't find out to do it easily. For now, I use these ones:

Code Block

	none
	none

  class Frame(object):
    def __init__(self, name="", title=""):
    def myticks(self, x, pos):
    def show(self):
    def update_axes(self):
    #def set_ticks_new(self, nticks, lotick, hitick, axis="X"):
    def set_ticks(self, limits = None ):

  class Plotter(object):
    def __init__(self):
    def add_frame(self, name="", title="",contents=None, type=None, aspect='auto'):
    def plot_all_frames(self, fignum=1, ordered=False):
    def settings(self
                 , width = 8 # width of a single plot
                 , height = 7 # height of a single plot
                 , nplots=1  # total number of plots in the figure
                 , maxcol=3  # maximum number of columns
                 ):
    def create_figure(self, fignum, nplots=1):
    def close_figure(self):
    def connect(self,plot=None):
    def onpick(self, event):
    def onclick(self, event) :
    def plot_image(self, image, fignum=1, title="", showProj = False, extent=None):
    def plot_several(self, list_of_arrays, fignum=1, title="" ):
    def draw_figurelist(self, fignum, event_display_images, title="",showProj=False,extent=None ) :
    def draw_figure( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None):
    def drawframe( self, frameimage, title="", fignum=1,position=1, showProj = False,extent=None):

Anchor

	IPython
	IPython

Interactive plotting with IPython

Since XtcExplorer V00-01-04, you also have the option of launching ipython at the end of the job. The basic support for this has been added to all modules. Each pyana module stores some data (primarily as numpy arrays) which it passes to the pyana_plotter module which launches iPython. In the session you can investigate the data further and make additional plots.

Do 'whos' to see the workspace/namespace contents.
Exit the ipython session with Ctrl-D when you are done.
you can save the workspace using the ipython 'save'

Here are a few screenshot examples:

Full screenshot of all GUIs, plots and ipython session:
Image Added

A couple of close-ups of the ipython terminal:
Image Added Image Added

xtcscanner

This tool also belongs to the XtcExplorer package, and is used by the GUI. But the tool can also be run directly from the command line:

Code Block
usage: xtcscanner [options] xtc-files ... options: -h, --help show this help message and exit -n NDATAGRAMS, --ndatagrams=NDATAGRAMS -v, --verbose -l L1_OFFSET, --l1-offset=L1_OFFSET

Example:

Code Block

	none
	none
title	xtcscanner -n 200 /reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s0*

Scanning....
Start parsing files:
['/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s00-c00.xtc', '/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s01-c00.xtc']
  201 datagrams read in 0.070000 s .   .   .   .   .   .   .
-------------------------------------------------------------
XtcScanner information:
  - 1 calibration cycles.
  - Events per calib cycle:
   [197]

Information from  0  control channels found:
Information from  9  devices found
                      BldInfo:EBeam:             EBeamBld (197)
            BldInfo:FEEGasDetEnergy:             FEEGasDetEnergy (197)
        DetInfo:AmoETof-0|Acqiris-0:  (5 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
      DetInfo:AmoGasdet-0|Acqiris-0:  (2 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoITof-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoMbes-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
     DetInfo:EpicsArch-0|NoDevice-0:             Epics_V1 (688)
         DetInfo:NoDetector-0|Evr-0:             EvrConfig_V2 (1)
                          ProcInfo::             RunControlConfig_V1 (11)
XtcScanner is done!
-------------------------------------------------------------

Further analysis with `pyana`

Any serious data analysis will need more customized tools than we can provide in this general GUI interface.

This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python.
"Blank" analysis code can be generated with Andy's codegen script (try codegen -h and codegen -p for options). Extensive information about pyana is available on confluence.

Running this package's pyana modules outside of the XtcExplorer

Feel free to use any or all of the XtcExplorer modules as examples or starting point for your analysis. If you have suggestions for features or algorithms that might benefit others too, let me know and perhaps we can add it to the XtcExplorer package.

Example how to run a pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module -m XtcExplorer.pyana_ploter <xtc files>

Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots, you can also do without the pyana_plotter module:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module <xtc files>

Submitting to batch

Once you've identified the command line to run your pyana job, you can submit this to the LFS batch queue system as described in Computing (including Analysis).

Be aware that the pyana scripts used by the explorer use matplotlib QtAgg backend to display plots, and this will fail if you submit it to batch processing. While I look for a better solution, you can do the following to work around this problem:

Configure pyana to plot as little as possible by setting the display mode to NoDisplay
Add the following two lines to XtcExplorer/src/utilities.py :
import matplotlib
matplotlib.use('PDF')

A few words on time and performance

The explorer hasn't been optimized for speed properly, but I strive to keep the pyana examples relatively efficient. But keep in mind
that plotting with matplotlib does take a substansial amount of time. An example:

Running the explorer on a small xtc file from a XCS commissioning run, containing 1267 events, and plotting information
from two IPIMB detectors and a YAG camera image.

Processing all 1267 events and plotting only at the end took 7.26 s
Processing all 1267 events, plotting (SlideShow mode) at every 100 events took 25.6 s
Processing all 1267 events, plotting (SlideShow mode) at every 10 events took 197.0 s

So matplotlib drawing these plots take about 1.5 s each time, quite substantial compared to 5.7 microseconds for pyana to process each event. Other types of plots may take longer to plot, e.g. larger images, waveforms, etc.

Image AddedMore information about pyana can be found on confluence.

Page tree

Versions Compared

Old Version 113

New Version Current

Key

xtcexplorer (GUI interface to pyana)

How to get started

Description of the GUIs

LCLS Xtc Explorer

Pyana Control Center

Description of the GUIs

LCLS XTC Explorer

Pyana Control Center

The pyana modules

Common configuration parameters

pyana_plotter

pyana_bld

pyana_waveform

pyana_cspad

Common configuration parameters

pyana_image.py

pyana_plotter

pyana_bld

pyana_waveform

pyana_ipimb

pyana_epics

pyana_scan

xtcscanner

pyana_image

Further analysis with `pyana`

pyana_ipimb

pyana_epics

pyana_scan

Helper functions in utilities.py

Interactive plotting with IPython

xtcscanner

Further analysis with `pyana`

Running this package's pyana modules outside of the XtcExplorer

Submitting to batch

A few words on time and performance

Page tree

Page History

Versions Compared

Old Version 113

New Version Current

Key

xtcexplorer (GUI interface to pyana)

How to get started

Description of the GUIs

LCLS Xtc Explorer

Pyana Control Center

Description of the GUIs

LCLS XTC Explorer

Pyana Control Center

The pyana modules

Common configuration parameters

pyana_plotter

pyana_bld

pyana_waveform

pyana_cspad

Common configuration parameters

pyana_image.py

pyana_plotter

pyana_bld

pyana_waveform

pyana_ipimb

pyana_epics

pyana_scan

xtcscanner

pyana_image

Further analysis with pyana

pyana_ipimb

pyana_epics

pyana_scan

Helper functions in utilities.py

Interactive plotting with IPython

xtcscanner

Further analysis with pyana

Running this package's pyana modules outside of the XtcExplorer

Submitting to batch

A few words on time and performance

Further analysis with `pyana`

Further analysis with `pyana`