• Created by Unknown User (ofte), last modified on Jun 15, 2011

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 147 Next »

XtcExplorer (GUI interface to pyana)

This is a description and documentation of a GUI tool to explore the xtc data. It is written in python, relying on PyQt4 for graphical user interface. The data processing is done via the pyana framework and visualization provided by matplotlib. It can be run on any xtc file, including the file being written during data aquisition (DAQ).

Xtc files contain the raw data (datagrams) streamed from the DAQ online system. The datagrams occur in the order they were flushed from the DAQ system, and thus do not necessarily appear in chronological order. The datagrams are not indexed, so the file must be read sequentially. The pyana and psana frameworks are designed to process these datagrams and present them to you shot by shot (or event by event, in pyana/psana terminology).

The XtcExplorer allows a simple-to-run interface to the xtc files. The package name is XtcExplorer, the executables (xtcexplorer and xtcscanner) are found in the app subdirectory of this package, and all other source code is in the src subdirectory.

For a similar tool to explore the HDF5 data format, look at HDF5 Explorer.

How to get started

  1. Set up your release work directory:
    You should always run this code from your own release working directory.
    See the Account Setup section to set up the analysis environment. Set up your release working directory from pslogin: 
    [user@pslogin ~] newrel ana-current myrelease
[user@pslogin ~] cd myrelease
[user@pslogin myrelease] sit_setup

  2. Add the package:
    This step is optional. Since release ana-0.3.7, xtcexplorer has been built into the release so you can skip to (3).
    Checking out a local copy is beneficial, though! That will enable you to edit the source files and customise the analysis.

    You need an afs token to check out the package, so if you didn't already you need to run 'kinit' first: 
    [user@pslogin myrelease] kinit
Password for user@SLAC.STANFORD.EDU:
    Now you're ready to check out the package and "compile": 
    [user@pslogin myrelease] addpkg XtcExplorer V00-01-05
[user@pslogin myrelease] scons
    Note! If you omit the "tag" (VXX-XX-XX), you'll get the same version that's built into the release.
    Sometimes a more recent version exists. To get the latest and greatest version in the svn repository,
    instead do 'addpkg XtcExplorer HEAD', but be aware that it may look different from described here, and
    might have fresh bugs as well as new features.

  3. Run the program:
    Launch the GUI with the command 'xtcexplorer' and optionally give the input xtc files that you want to read
    as arguments. You can also browse to find files after launching the browser.

    To have access to the xtc files, you need to login to psana for this: 
    [user@psanaXXX myrelease] sit_setup
[user@psanaXXX myrelease] xtcexplorer /reg/d/psdm/CXI/cxi22010/xtc/e60-r0465-s0*
  • If you encounter any problems, ask for help! Don't be stuck. This is supposed to be easy. If it's not, I did something wrong, so let me know! (smile) Send me an email (ofte at slac.stanford.edu)

  • xtcexplorer writes out pyana configuration files, based on selections made in the GUI. Look out for files with names xb_pyana_XXXX.cfg, where XXXX is a random four-digit number. You can use them to run pyana directly from the command line. But they accumulate, so you might want to remove these regularly...

  • LCLS data is very diverse, and not all features for all detectors are implemented (yet). Feel free to contact me if you have requests and suggestions.

Description of the GUIs

LCLS Xtc Explorer

xtcexplorer will launch the main GUI. At startup, it has two sections:

  • File section: Shows a list of currently selected file(s). Please be adviced that you should only select a single run at a time (or combine runs where you know the setup / configuration is the same). A single run is typically spread over several XTC files (divided into streams (-s0X) and chunks (-c0X)).
    A "Clear File List" removes all filenames from the list.
    You have three options for adding files:
    1. You can supply file names at the command line when launching the application.
    2. A "File Browser" button opens a file browser, where you can select one or more files.
    3. Or you can type file name by hand in the input line.
  • Scan section: Once you are done with selecting files, you should push one of the "Scan" buttons to connect and scan the files. If the file is large, the "Scan File(s)" button will be disabled. Push the "Quick Scan" to get a summary of the file contents.
    Note, for most purposes, a "Quick Scan" is sufficient. If you need to scan the whole file, e.g. if you want to know the total number of events, number of calibration cycles, etc, you can enable the "Scan File(s)" button.

    Once the file(s) have been scanned, a new GUI will pop up and show information about the file content and help with further processing of the data in pyana.

The main window before any file selection. Click on "File Browser..." to select file(s).

A standard file browser pops up. Navigate the file browser till you find the xtc files for your experiment. (Remember, if this seems cumbersome, you can also give the file name(s) as argument to xtcexplorer when you first launch the GUI)

Mark the files you want to look at (Hold down the Ctrl-key to select more than one file). Note, you should only look at one run number at a time, since different runs might have different configurations.

This is the main window after a file has been selected. File name and file size is shown in the GUI. If the file is not too big, you can click the "Scan File(s)" button to get exact contents of the whole file. If the file is big, it's better to do a "Quick Scan" which will tell you all you need to know (except count number of events and calibration cycles).

Pyana Control Center

After scanning, a new GUI will pop up showing you a list of detectors/devices found in the file. A little more information is written to the terminal window too.

Main window after the file scan. Displays the number of events and calibration cycles (scan steps)

Another window, the Pyana Control Center, pops up after the file scan.

The pyana control center has a field to the left that shows you the detector/device sources found within the file(s). In front of each detector/device name is a checkbox, where you can select which datagrams you are interested in analysing / plotting.

To the right of this is a field with information and general run options and default plotting options: e.g. how many events to process total, skip events at the beginning of the file(s), How often to display plot (default is every 100 events, unless several scan steps, in which case it will plot for each new scan step). Also allows you to select display mode (default is "SlideShow") and opt to launch iPython at the end for interactive investigation of the data

Some more information is printed to the terminal window from which the xtcbrowser was launched

Once you checkmark the detectors you want to display information from, another tab will pop up showing pyana configuration text. "Current pyana configuration": as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field.

If a ControlPV is present and checked off, only a pyana_scan module will be used. All the other devices you check will be added to the input of the scan.
If no ControlPV is used, other pyana modules will be configured as appropriate to display a variety of information from the events.
If "Epics Process Variables" are checked off, another Gui appears that lists all the epics variables. Select the ones you want to display.

Press the "Write configuration to file" button once you're done. Once a file is written, a "Run pyana" button will appear.

You can further edit the configuration file by hand. If you click on "Edit configuration", the file will open in the default editor. Note! The default editor is defined in your EDITOR environment variable. Make sure it is set to your editor of choice (e.g. setenv EDITOR emacs).

"Run pyana" lauches an input GUI that shows you the runstring. You can use the same runstring from the command line. Or hit "OK" and it'll run.

After launching pyana, another button "Quit pyana" appears... If you see you need to change parameters, you can stop pyana, edit the configuration file, and start over again.

More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).

The pyana modules

The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!

XtcExplorer/src/pyana_bld.py          # display of Beam-line data
XtcExplorer/src/pyana_epics.py        # display of Epics PV data
XtcExplorer/src/pyana_scan.py         # display of motor scan data
XtcExplorer/src/pyana_encoder.py      # display of encoder data
XtcExplorer/src/pyana_ipimb.py        # display of diode data from IPIMB and PIM
XtcExplorer/src/pyana_waveform.py     # display of waveform data
XtcExplorer/src/pyana_cspad.py        # display of CsPad image data
XtcExplorer/src/pyana_image.py        # display of camera image data
XtcExplorer/src/pyana_plotter.py      # a plotter module to control the event display

Additional helper classes and functions are defined in 

XtcExplorer/src/utilities.py          # PyanaOptions                              - help parse options
                                      # BaseData and classes derived thereof      - store numpy arrays for ipython
                                      # Threshold                                 - keep track of threshold values and region of interest
                                      # Frame                                     - container for a single "plot"
                                      # Plotter                                   - container for a single "canvas"

XtcExplorer/src/cspad.py              # CsPad                                     - for arranging & aligning the CsPad elements

These are imported into the pyana modules wherever needed. You might want to keep this in mind when
using examples from this package. The pyana modules can mostly be run on their own, but they might
need to import this module.

A few things to note about the different detectors / pyana modules:

Common configuration parameters

A few parameters are common to all modules, and their default values are set to be the same, but can also be set individually for each module. They are the following:

# parameter name   default value   comment
  plot_every_n      = 100         # If 0, plot only at the end. Else, display every N events
  accumulate_n      = 0           # If 0, accumulate all (no reset). Else, reset stored arrays every N events
  fignum            = 1           # matplotlib figure number

pyana_plotter

This module is added to the end of the job, after other modules. It controls the display mode and ipython.

  • The default display mode is SlideShow mode, where the plot window stays open but updates every n events. In Interactive mode, the figure window must be closed before the next event gets processed. The other modules are set up to work with this one, but should with minor modifications also run fine without it. (The minor modifications would be to issue a "plt.ion()" before the figures are created or replace "plt.draw()" with "plt.show()" to make sure the plot windows stay open.)
  • There's also a handle to allow you to use ipython at the end of the job (or after each plot) to investigate your plottet arrays further. 
      display_mode      = 2           #  Interactive (1) or SlideShow (2) or NoDisplay (0)
  ipython           = False       #  Drop into ipython after each plot (to allow further investigation of data)

pyana_bld

This module collects data from the BeamLine Detectors (Bld) and plots them in separate windows. 

 do_ebeam           = False       #  Plot data from EBeam object
 do_gasdetector     = False       #  Plot data from GasDetector
 do_phasecavity     = False       #  Plot data from PhaseCavity
 do_ipimb           = False       #  Plot data from SharedIpimb

Example of how to run this pyana module by itself, i.e. no GUI, processing 300 shots and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_bld -m XtcExplorer.pyana_plotter <xtc files>

pyana_waveform

Plots waveform data from Acqiris digitizers. Takes a list of addresses, currently will make one plot for each channel of each device. Plan to add more options.

Configurable options:

  sources           =             # string, space-separated list of addresses (Detector-Id|Device-ID)

Example how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform -m XtcExplorer.pyana_ploter <xtc files>

Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_waveform <xtc files>

pyana_cspad

CsPad data is reconstructed in pyana_cspad.py.

  • If a threshold is provided, shots above threshold will be displayed / stored, and shots below
    threshold will be added to sum of darks for averaging.
  • If a dark file is provided, background subtraction is done before filtering. If not, the below-threshold
    average can be used for background subtraction
  • The plot configuration depends on whether threshold and dark subtraction is used. Intensity projections
    are also added to the plot, can be turned off (in the code).
  • In Interactive mode (see pyana_plotter), you can click on the image to
    • change intensity range (color range): Left-click to set low limit, right-click to set high limit, middle-click to reset
    • change threshold value and region of interest: Click on the rectangle, enter new values when prompted in terminal window
    • switch to SlideShow mode when you are content with the display: Middle-click on the canvas and enter new display mode value.
  • You can save the average image(s) or the selected single-shot images to file (numpy binary file)

Options must be specified in a configuration file. Only the img_sources parameter is required.

  img_sources       =             # string, Address of Detector-Id|Device-ID
  dark_img_file     =             # filename of a binary numpy (.npy) with dark image for subtraction
  out_avg_file      =             # name (base) of output file for average image (numpy array)
  out_shot_file     =             # name (base) of numpy array file for selected single-shot events
  plot_vrange       =             # range (format vmin:vmax) of values for plotting (pixel intensity)
  threshold         =             # threshold intensity and threshold area. Format: value (xlow:xhigh,ylow:yhigh)

CsPad image where the region of interest (a red rectangle in the left figure) has intensity above a given threshold. The right image shows the region of interest.

CsPad image where region of interest (red rectangle) has intensity above a given threshold. Additionally, a dark-subtracted image is shown (bottom left), where the dark average is based on below-threshold shots from the same run (shown bottom right.

This image is similar to the one above. The intensity limits have been adjusted interactively.

At the end of the job, the average of good images (left) and rejected / dark images (right) is shown.

Example of how to run this pyana module by itself, i.e. no GUI, processing 10 events and using an xtcexplorer-generated config file:

pyana -n 10 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_cspad -m XtcExplorer.pyana_plotter <xtc files>

pyana_image.py

This module processes generic camera frames, e.g from Pulnix TM6740 device. It allows any number of images, given as a space-separated list of addresses in the configuration file.

  • You can set ranges to define good images and dark images. If both are set, you have the option to display good images background subtracted, where background subtraction is based on the average of background images so far collected.
  • Each image can be separately rotated, shifted and scaled (zoomed in/out).
  • Nicknames can be given to the input images. Defaults are Im1, Im2... etc. These names will be used if you plot differences, or other manipulations of the original images.
  • The images are subtracted and differences displayed as well as fourier transform of differences. Examples of what may be displayed. To display other things, at this stage you have to edit pyana_image.py to change this behaviour. 
     sources             # address string of Detector-Id|Device-ID
 good_range          # threshold values selecting images of interest (Format: low,high)
 dark_range          # threshold values selecting dark images (Format: low,high)
 image_rotations     # rotation angle, in degrees, to be applied to image(s)
 image_shifts        # npixel shifts, format (nx,ny), to be applied to image(s)
 image_scales        # scale factor (float) to be applied to images
 image_nicknames     # nicknames for plot titles
 image_manipulations # String containing keywords: Diff, FFT
 output_file         # filename. Valid extensions are .hdf5, .txt (ascii) or .npy (numpy binary)
 n_hdf5              # if output file is hdf5, combine n events in each output file.

    Image from two Pulinx cameras, no filtering or background subtraction. Projections in x and y are shown.

    Displays of three different Pulnix TM6740 images of YAG screens, after rotation/translation. Also shown, differences between images and FFT of differences.

Example of how to run this pyana module by itself, i.e. no GUI, processing 10 events and using an xtcexplorer-generated config file:

pyana -n 10 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_image -m XtcExplorer.pyana_plotter <xtc files>

pyana_ipimb

This module extracts and displays data from IPIMB type detectors. There are two types of information: IpimbData (raw data) and IpmFex (feature extracted data). Only the last type of data is currently implemented, but other data can also be added. Currently the display will show a plot of the intensity measured in each of the four channels, the beam spot position estimated from sums and differences of the individual intensity measurements, and the sum of the four measurements as a function of time. The plot included below has these plots for 4 different IPIMB detector sources. 

ipimb_addresses     # list of IPIMB addresses

Example of how to run this pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_ipimb -m XtcExplorer.pyana_plotter <xtc files>

pyana_epics

pv                  # Name(s) of the EPICS PV(s) to dump

Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:

pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_epics -m XtcExplorer.pyana_plotter <xtc files>

pyana_scan

This module is different from the others. In runs with several calibration cycles and one (or more) control PV (process variable), this module displays scalar values from other detectors as a function of scan step of the control PV. It currently takes two scalar type inputs to evaluate the scan:

input_epics         # Name(s) of other scalars to correlate in scan
input_scalars       # Name(s) of other scalars to correlate in scan

In addition, the control PV name is added to the parameter list. Currently it is only for display, as the module picks up the active control PV withouth using this parameter. 

controlpv = None

Note! The XtcExplorer behaviour is different when you check off the ControlPV checkbox! In this case only the pyana_scan module is activated, and all other selected detectors are added as input to this module.

Example of how to run this pyana module by itself, i.e. no GUI, processing 200 events and using an xtcexplorer-generated config file:

pyana -n 200 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_scan -m XtcExplorer.pyana_plotter <xtc files>

Interactive plotting with IPython

Since XtcExplorer V00-01-04, you also have the option of launching ipython at the end of the job. The basic support for this has been added to all modules. Each pyana module stores some data (primarily as numpy arrays) which it passes to the pyana_plotter module which launches iPython. In the session you can investigate the data further and make additional plots.

  • Do 'whos' to see the workspace/namespace contents.
  • Exit the ipython session with Ctrl-D when you are done.
  • you can save the workspace using the ipython 'save'

Here are a few screenshot examples:

Full screenshot of all GUIs, plots and ipython session:

A couple of close-ups of the ipython terminal:

xtcscanner

This tool also belongs to the XtcExplorer package, and is used by the GUI. But the tool can also be run directly from the command line: 

usage: xtcscanner [options] xtc-files ...

options:
  -h, --help            show this help message and exit
  -n NDATAGRAMS, --ndatagrams=NDATAGRAMS
  -v, --verbose
  -l L1_OFFSET, --l1-offset=L1_OFFSET

Example:

xtcscanner -n 200 /reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s0*
Scanning....
Start parsing files:
['/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s00-c00.xtc', '/reg/d/psdm/AMO/amo01509/xtc/e8-r0094-s01-c00.xtc']
  201 datagrams read in 0.070000 s .   .   .   .   .   .   .
-------------------------------------------------------------
XtcScanner information:
  - 1 calibration cycles.
  - Events per calib cycle:
   [197]

Information from  0  control channels found:
Information from  9  devices found
                      BldInfo:EBeam:             EBeamBld (197)
            BldInfo:FEEGasDetEnergy:             FEEGasDetEnergy (197)
        DetInfo:AmoETof-0|Acqiris-0:  (5 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
      DetInfo:AmoGasdet-0|Acqiris-0:  (2 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoITof-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
        DetInfo:AmoMbes-0|Acqiris-0:  (1 ch)     AcqConfig_V1 (1)   AcqWaveform_V1 (197)
     DetInfo:EpicsArch-0|NoDevice-0:             Epics_V1 (688)
         DetInfo:NoDetector-0|Evr-0:             EvrConfig_V2 (1)
                          ProcInfo::             RunControlConfig_V1 (11)
XtcScanner is done!
-------------------------------------------------------------

Further analysis with pyana

Any serious data analysis will need more customized tools than we can provide in a this general GUI interface.

This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python.
"Blank" analysis code can be generated with Andy's codegen script (try codegen -h and codegen -p for options). Extensive information about pyana is available on confluence.

Feel free to use any or all of the XtcExplorer modules as examples or starting point for your analysis. If you have suggestions for features or algorithms that might benefit others too, let me know and perhaps we can add it to the XtcExplorer package.

Example how to run a pyana module by itself, i.e. no GUI, processing 300 events and using an xtcexplorer-generated config file:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module -m XtcExplorer.pyana_ploter <xtc files>

Or if you add "plt.ion()" to the constructor for SlideShow mode OR "plt.show()" to make_plot functions for interactive plots, you can also do without the pyana_plotter module:

pyana -n 300 -c xb_pyana_5299.cfg -m XtcExplorer.pyana_module <xtc files>

  • No labels