Event display / xtc file browser

Xtc files contain the raw data streamed from the DAQ online system, therefore they are not indexed and the events don't always line up in the "right" order. Therefore it's not straight-forward to browse (back and fourth) through an xtc file. This tool (XtcEventBrowser) is also not a real browser, but allows a simple-to-run interface to the xtc files. The package name is XtcEventBrowser, the executables (xtcbrowser and xtcscanner) are found in the app subdirectory of this package, and all other code is in the src subdirectory.

xtcbrowser

This page is currently under revision. See an earlier version (e.g. v77) of this page ("Tools"->"Page History") to get self-consistent documentation.

The xtcbrowser is the command to launch the Event Display for xtc files. The package name is XtcEventBrowser. It is written in python, relying on PyQt4 for graphical user interface. The data processing is done via the pyana framework and visualization provided by matplotlib.

Note! This tool is under development... features are being added and new versions available often. This documentation might be slightly outdated, but if you use the tagged version (VXX-XX-XX) mentioned here it should work as advertised.

How to get started

To run, you need to set up an offline release in your directory (See the Account Setup section to set up the analysis environment):

[user@psana0XXX ~] newrel ana-current myrelease
[user@psana0XXX ~] cd myrelease
[user@psana0XXX ~] sit_setup

Add the xtc browser package to your analysis release and "compile":

[user@psana0XXX myrelease] addpkg XtcEventBrowser V00-00-16
[user@psana0XXX myrelease] scons

Note! You can omit the "tag" (VXX-XX-XX) to get the latest version in the svn repository, but this may look different from described here.

Run the program with the command 'xtcbrowser' and optionally give the input xtc files that you want to read as arguments. You can also browse to find files after launching the browser.

[user@psana0XXX myrelease] xtcbrowser /reg/d/psdm/cxi/cxi80410/xtc/e55-r0581*

Description of the GUIs

LCLS Xtc Event Browser

The procedure above will open a GUI, the main browser. It allows you to browse for files, and to run a scan to see what's in the file. Maybe "scan" is not a good choice of word... it parses the xtc file and investigate what kind of data is there.

• File section: Shows a list of currently selected file(s). As you may have guessed, "File Browser" opens a file browser and "Clear File List" clears the current list of files. This section also allows you to add a file name by hand (or paste).
• Scan section: The two buttons to the left allows you to scan the xtc file to get a summary of what datagrams are stored in it. Note, for most purposes, a "Quick Scan" is sufficient. If you need to scan the whole file, e.g. if you want to know the total number of events, number of calibration cycles, etc, you can enable the "Scan File(s)" button. If the files are big, this will take a lot of time...

Pyana Control Center

After scanning, a new GUI will pop up showing you a list of detectors/devices found in the file. A little more information is written to the terminal window too.

• "Available Detectors/Devices": In front of each detector/device name is a checkbox, where you can select which datagrams you are interested in analysing / plotting.
• "Current pyana configuration": Initially this field is blank. But as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field. At the same time, another two buttons shows ut:
• "Write configuration to file" and "Edit configuration file". You need to write the configuration to file to be able to run pyana (which picks up this file).
• "Run pyana" button will appear once you've written to file. You can still edit the file (which lauches an emacs window... my apologies to non-emacs-users... Will have a more generic solution soon'ish). "Run pyana" lauches an input GUI that shows you the runstring. You can use the same runstring from the command line. Or hit "OK" and it'll run.
• After launching pyana, another button "Quit pyana" appears... If you see you need to change parameters, you can stop pyana, edit the configuration file, and start over again.

More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).

Example plots: one event from CsPad with background subtraction and filter. Some beamline data plots: Beam energy and position, Gas detector energy measurements.

A few things to note about the different detectors / pyana modules:

• CsPad: CsPad data is reconstructed in pyana_cspad.py. The image plot value limits are adjusted automatically, but if
  you want to change them, click on the color bar (left-click for low limit, right-click for high limit).
  The successive events will be plotted with the new limits. Revert to the original by middle-clicking.
  To run this module by itself with pyana:

  pyana -m XtcEventBrowser/src/pyana_cspad.py <xtc files>
  

  Options must be specified in a configuration file, or the default values will be used, e.g.:

  image_source    =  CxiDs1-0|Cspad-0   # string, Address of Detector-Id|Device-ID
  draw_each_event =                     # bool, Draw plot for each event? (Default=False).
  dark_img_file   =                     # filename, Dark image file to be loaded, if any
  plot_vrange     =                     # range=vmin-vmax (intensity) to be plotted, default is full range
  threshold       =                     # lower threshold for image intensity in threshold area of the plot
  thr_area        =                     # range=xmin,xmax,ymin,ymax defining threshold area
  output_file     =                     # filename for saving numpy array with average of images
  

• pyana_image.py processes generic camera frames, e.g from Pulnix TM6740 device. It allows any number of images, given as a space-separated list of addresses in the
  configuration file.
  • You can set ranges to define good images and dark images. If both are set, you have the option to display good images background subtracted, where background subtraction is based on the average of background images so far collected.
  • Each image can be separately rotated, shifted and scaled (zoomed in/out).
  • Nicknames can be given to the input images. Defaults are Im1, Im2... etc. These names will be used if you plot differences, or other manipulations of the original images.
  • The images are subtracted and differences displayed as well as fourier transform of differences. Examples of what may be displayed. To display other things, at this stage you have to edit pyana_image.py to change this behaviour.

• Currently it has the following settings:

  image_addresses  =  CxiSc1-0|TM6740-1 CxiSc1-0|TM6740-2 CxiSc1-0|TM6740-3 # Address of Detector-Id|Device-Id
  dark_range  =  50--250                # low and high limit for what we define as dark image
  good_range  =  250--1050              # low and high limit for what we define as a good image (with signal)
  image_rotations  =  7.1 6.2 5.3       # Angle in degrees
  image_shifts  =  (0,0) (0,0) (0,0)    # Shift (number of pixels (x,y)) to be applied
  image_scales  =                       # Scale factor to be applied to zoom in or out
  image_nicknames = Im1 Im2 Im3         # If none provided, these will be the names
  image_manipulations =                 # String containing keywords: "Diff" for difference plots, FFT for FFT of difference arrays
  draw_each_event  =  Yes               # plot for each event?
  output_file = myarrays.hdf5           # base name for output file. Valid extensions are .hdf5, .txt (ascii) or .npy (numpy binary)
                                        # numpy arrays can only be written one per file.
  n_hdf5                                # if HDF5 output, this parameter allows you to split the output with N events in each
  

xtcscanner

This is a command-line interface to the XtcScanner class that makes a summary of the xtc file.

usage: xtcscanner [options] xtc-files ...

options:
  -h, --help            show this help message and exit
  -n NDATAGRAMS, --ndatagrams=NDATAGRAMS
  -v, --verbose
  -l L1_OFFSET, --l1-offset=L1_OFFSET

Further analysis with pyana

Any serious data analysis will need more customized tools than we can provide in a GUI interface. This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python. The Gui Event Browser can provide simple analysis code that can be expanded by the user. "Blank" analysis code can also be generated with Andy's codegen script (try codegen -h and codegen -p for options).

More information about pyana can be found on confluence.

Data visualization with NumPy (arrays) and MatPlotLib (plots).

Saving (and loading) a numpy array (e.g. image) to (from) a file

If you want to save one array (max 2 dimensions), you can use binary numpy file or ascii file:

import numpy as np

# binary file .npy format
np.save("filename.npy", array)
array = np.load("filename.npy")

# txt file
np.savetxt("filename.dat", array)
array = loadtxt("filename.dat")

If you need to save multiple events/shots in the same file you will need to do some tricks (e.g. flatten the array and stack 1d arrays into 2d arrays where axis2 represent event number). Or you could save as an HDF5 file.

You can save an array or several into an HDF5 file (example from pyana):

import h5py

def beginjob(self,evt,env):
    self.ofile = h5py.File("outputfile.hdf5", 'w') # open for writing (overwrites existing file)
    self.shot_counter = 0

def event(self,evt,env)
    # example: store several arrays from one shot in a group labeled with shot (event) number
    self.shot_counter += 1
    group = self.ofile.create_group("Shot%d" % self.shot_counter)

    image1_source = "CxiSc1-0|TM6740-1"
    image2_source = "CxiSc1-0|TM6740-2"

    frame = evt.getFrameValue(image1_source)
    image1 = frame.data()
    frame = evt.getFrameValue(image2_source)
    image2 = frame.data()

    dataset1 = group.create_dataset("%s"%image1_source,data=image1)
    dataset2 = group.create_dataset("%s"%image2_source,data=image2)

def endjob(self,env)
    self.ofile.close()

Or you can group your datasets any other way you find useful, of course.

A comparison with MatLab.

%
%
%
a1 = subplot(121);
loglog(channels(:,1),channels(:,2),'o')
xlabel('CH0')
ylabel('CH1')
a2 = subplot(122);
loglog(channels(:,3),channels(:,4),'o')
xlabel('CH2')
ylabel('CH3')

import matplotlib.pyplot as plt
import numpy as np

a1 = plt.subplot(221)
plt.loglog(channels[:,0],channels[:,1], 'o' )
plt.xlabel('CH0')
plt.ylabel('CH1')
a2 = plt.subplot(222)
plt.loglog(channels[:,2],channels[:,3], 'o' )
plt.xlabel('CH2')
plt.ylabel('CH3')

axes(a1)
hold on
lims(1:2,:) = ginput(2);

axes(a2)
hold on
lims(3:4,:) = ginput(2);

plt.axes(a1)
plt.hold(True)
limslista = plt.ginput(2)

plt.axes(a2)
plt.hold(True)

limslistb = plt.ginput(2)
limsa = np.array(limslista)
limsb = np.array(limslistb)

lims = np.hstack( [limsa, limsb] )

fbool1 = (channels(:,1)>min(lims(1:2,1)))&(channels(:,1)<max(lims(1:2,1)))
fbool2 = (channels(:,2)>min(lims(1:2,2)))&(channels(:,2)<max(lims(1:2,2)));
fbool = fbool1&fbool2
loglog(channels(fbool,1),channels(fbool,2),'or')

fbool3 = (channels(:,3)>min(lims(3:4,3)))&(channels(:,3)<max(lims(3:4,3)))
fbool4 = (channels(:,4)>min(lims(3:4,4)))&(channels(:,4)<max(lims(3:4,4)));
fbool = fbool3&fbool4
loglog(channels(fbool,3),channels(fbool,4),'or') 

fbools0 = (channels[:,0]>lims[:,0].min())&(channels[:,0]<lims[:,0].max())
fbools1 = (channels[:,1]>lims[:,1].min())&(channels[:,1]<lims[:,1].max())
fbools = fbools0 & fbools1

fbools2 = (channels[:,2]>lims[:,2].min())&(channels[:,2]<lims[:,2].max())
fbools3 = (channels[:,3]>lims[:,3].min())&(channels[:,3]<lims[:,3].max())
fbools = fbools2&fbools3