Event display / xtc file browser
Xtc files contain the raw data streamed from the DAQ online system, therefore they are not indexed and the events don't always line up in the "right" order. Therefore it's not straight-forward to browse (back and fourth) through an xtc file. This tool (XtcEventBrowser) is also not a real browser, but allows a simple-to-run interface to the xtc files. The package name is XtcEventBrowser, the executables (xtcbrowser and xtcscanner) are found in the app subdirectory of this package, and all other code is in the src subdirectory.
xtcbrowser
The xtcbrowser is the command to launch the Event Display for xtc files. The package name is XtcEventBrowser. It is written in python, relying on PyQt4 for graphical user interface. The data processing is done via the pyana framework and visualization provided by matplotlib.
Note! This tool is under development... features are being added and new versions available often. This documentation might be slightly outdated, but if you use the tagged version (VXX-XX-XX) mentioned here it should work as advertised.
How to get started
This tool is not in a release yet, so to run, you need to set up an offline release in your directory (See the Account Setup section to set up the analysis environment):
[user@psana0XXX ~] newrel ana-current myrelease [user@psana0XXX ~] cd myrelease [user@psana0XXX ~] sit_setup
Add the xtc browser package to your analysis release and "compile":
[user@psana0XXX myrelease] addpkg XtcEventBrowser V00-00-18 [user@psana0XXX myrelease] scons
Note! You can omit the "tag" (VXX-XX-XX) to get the latest version in the svn repository, but this may look different from described here.
Run the program with the command 'xtcbrowser' and optionally give the input xtc files that you want to read as arguments. You can also browse to find files after launching the browser.
[user@psana0XXX myrelease] xtcbrowser /reg/d/psdm/cxi/cxi80410/xtc/e55-r0581*
If you encounter any problems, ask for help! Don't be stuck. This is supposed to be easy. If it's not, I did something wrong, so let me know! Send me an email (ofte at slac.stanford.edu)
Description of the GUIs
LCLS Xtc Event Browser
xtcbrowser will launch a GUI, the main browser. It allows you to browse for files, and to run a scan to see what's in the file. (Perhaps "scan" is not a good choice of word... it parses the xtc file and investigate what kind of data is there.)
- File section: Shows a list of currently selected file(s). As you may have guessed, "File Browser" opens a file browser and "Clear File List" clears the current list of files. This section also allows you to add a file name by hand (or paste).
- Scan section: The two buttons to the left allows you to scan the xtc file to get a summary of what datagrams are stored in it. Note, for most purposes, a "Quick Scan" is sufficient. If you need to scan the whole file, e.g. if you want to know the total number of events, number of calibration cycles, etc, you can enable the "Scan File(s)" button. If the files are big, this will take a lot of time...
|
Main window before any file selection. |
|
Main window after a file has been selected. File name and file size is shown in the GUI. If the file is not too big, you can click the "Scan File(s)" button to get exact contents of the whole file. If the file is big, it's better to do a "Quick Scan" which will tell you all you need to know (except count number of events and calibration cycles). |
Pyana Control Center
After scanning, a new GUI will pop up showing you a list of detectors/devices found in the file. A little more information is written to the terminal window too.
|
Main window (top) after the file scan. In this case the file contain several "calibration cycles" (motor scan steps), and the GUI lists number of calibration cycles and number of events. Some more information is printed to the terminal window from which the xtcbrowser was launched. |
|
Once you checkmark the detectors you want to display information from, another tab will pop up showing pyana configuration text. "Current pyana configuration": as you select devices from the list, a tentative configuration file for running pyana is written and shown in this field. |
More information on how to run pyana by itself (see 'pyana -h' for more help, or the pyana section of confluence).
The pyana modules
The GUI as described above prepares a configuration file for you to run pyana. You can either run it from the GUI or you can run it from the command line. The configfile sets input parameters for the pyana modules in this package. Feel free to use one or more of these modules as a starting point for your more elaborate pyana analysis!
XtcEventBrowser/src/pyana_bld.py # display of Beam-line data XtcEventBrowser/src/pyana_cspad.py # display of CsPad image data XtcEventBrowser/src/pyana_image.py # display of camera image data XtcEventBrowser/src/pyana_ipimb.py # display of diode data from IPIMB and PIM XtcEventBrowser/src/pyana_waveform.py # display of waveform data (not fully functional yet) XtcEventBrowser/src/pyana_epics.py # display of Epics PV data (not fully functional yet) XtcEventBrowser/src/pyana_scan.py # display of motor scan data XtcEventBrowser/src/pyana_plotter.py # a plotter module to control the event display
A few things to note about the different detectors / pyana modules:
- A few configuration parameters are common to all modules, and their default values are set to be the same, but can also be set differently.
plot_every_n = 10 # If 0, don't plot till the end of the job. Else, display every N events fignum = 1 # "base" number for matplotlib figure numbering scheme
- pyana_plotter: This module is added to the end of the job, after other modules. It does nothing (yet) than control the display mode. Default is SlideShow mode.
display_mode = 2 # Interactive (1) or SlideShow (2) or NoDisplay (0)
- pyana_scan: This module is different from the others. It does a "motor scan", displays certain values as a function of scan step of a motor scan. The motor name is given in the xtc file as the Control PV. There may be more than one control PV. It currently takes two scalar type inputs to evaluate the scan:
input_epics # Name(s) of other scalars to correlate in scan input_scalars # Name(s) of other scalars to correlate in scan
- pyana_cspad: CsPad data is reconstructed in pyana_cspad.py. The image plot value limits are adjusted automatically, but if
you want to change them, click on the color bar (left-click for low limit, right-click for high limit).
The successive events will be plotted with the new limits. Revert to the original by middle-clicking.
To run this module by itself with pyana:Options must be specified in a configuration file, or the default values will be used, e.g.:pyana -m XtcEventBrowser/src/pyana_cspad.py <xtc files>
image_source # string, Address of Detector-Id|Device-ID dark_img_file # filename, Dark image file to be loaded, if any output_file # filename (If collecting: write to this file) plot_vrange # range=vmin-vmax of values for plotting (pixel intensity) threshold # lower threshold for image intensity in threshold area of the plot thr_area # range=xmin,xmax,ymin,ymax defining threshold area
Image display of the CSPad detector, background subtracted. This is currently the only display that has interactive features. A filter allows you to select events within requested intensity range.
- pyana_image.py processes generic camera frames, e.g from Pulnix TM6740 device. It allows any number of images, given as a space-separated list of addresses in the
configuration file.- You can set ranges to define good images and dark images. If both are set, you have the option to display good images background subtracted, where background subtraction is based on the average of background images so far collected.
- Each image can be separately rotated, shifted and scaled (zoomed in/out).
- Nicknames can be given to the input images. Defaults are Im1, Im2... etc. These names will be used if you plot differences, or other manipulations of the original images.
- The images are subtracted and differences displayed as well as fourier transform of differences. Examples of what may be displayed. To display other things, at this stage you have to edit pyana_image.py to change this behaviour.
image_addresses # address string of Detector-Id|Device-ID good_range # threshold values selecting images of interest (Format: low--high) dark_range # threshold values selecting dark images (Format: low--high) image_rotations # rotation angle, in degrees, to be applied to image(s) image_shifts # npixel shifts, format (nx,ny), to be applied to image(s) image_scales # scale factor (float) to be applied to images image_nicknames # nicknames for plot titles image_manipulations # String containing keywords: Diff, FFT output_file # filename. Valid extensions are .hdf5, .txt (ascii) or .npy (numpy binary) n_hdf5 # if output file is hdf5, combine n events in each output file.
Displays of three different Pulnix TM6740 images of YAG screens, after rotation/translation. Also shown, differences between images and FFT of differences.
- pyana_ipimb
ipimb_addresses # list of IPIMB addresses
- pyana_epics
pv # Name(s) of the EPICS PV(s) to dump
- pyana_bld
Beam energy and position. Gas detector energy measurements
do_ebeam # Plot data from EBeam object do_gasdetector # Plot data from GasDetector do_phasecavity # Plot data from PhaseCavity
xtcscanner
This is a command-line interface to the XtcScanner class that makes a summary of the xtc file.
usage: xtcscanner [options] xtc-files ... options: -h, --help show this help message and exit -n NDATAGRAMS, --ndatagrams=NDATAGRAMS -v, --verbose -l L1_OFFSET, --l1-offset=L1_OFFSET
Further analysis with pyana
Any serious data analysis will need more customized tools than we can provide in a GUI interface. This will require the user / analyst to program his/her own tools. Pyana is a complete framework for programming a user analysis in python. The Gui Event Browser can provide simple analysis code that can be expanded by the user. "Blank" analysis code can also be generated with Andy's codegen
script (try codegen -h
and codegen -p
for options).
More information about pyana can be found on confluence.
Data visualization with NumPy (arrays) and MatPlotLib (plots).
Saving (and loading) a numpy array (e.g. image) to (from) a file
If you want to save one array (max 2 dimensions), you can use binary numpy file or ascii file:
import numpy as np # binary file .npy format np.save("filename.npy", array) array = np.load("filename.npy") # txt file np.savetxt("filename.dat", array) array = loadtxt("filename.dat")
If you need to save multiple events/shots in the same file you will need to do some tricks (e.g. flatten the array and stack 1d arrays into 2d arrays where axis2 represent event number). Or you could save as an HDF5 file.
You can save an array or several into an HDF5 file (example from pyana):
import h5py def beginjob(self,evt,env): self.ofile = h5py.File("outputfile.hdf5", 'w') # open for writing (overwrites existing file) self.shot_counter = 0 def event(self,evt,env) # example: store several arrays from one shot in a group labeled with shot (event) number self.shot_counter += 1 group = self.ofile.create_group("Shot%d" % self.shot_counter) image1_source = "CxiSc1-0|TM6740-1" image2_source = "CxiSc1-0|TM6740-2" frame = evt.getFrameValue(image1_source) image1 = frame.data() frame = evt.getFrameValue(image2_source) image2 = frame.data() dataset1 = group.create_dataset("%s"%image1_source,data=image1) dataset2 = group.create_dataset("%s"%image2_source,data=image2) def endjob(self,env) self.ofile.close()
Or you can group your datasets any other way you find useful, of course.
A comparison with MatLab.
MatLab |
MatPlotLib |
Comments |
|
---|---|---|---|
Loglog plot of one array vs. another % % % a1 = subplot(121); loglog(channels(:,1),channels(:,2),'o') xlabel('CH0') ylabel('CH1') a2 = subplot(122); loglog(channels(:,3),channels(:,4),'o') xlabel('CH2') ylabel('CH3') |
Loglog plot of one array vs. another import matplotlib.pyplot as plt import numpy as np a1 = plt.subplot(221) plt.loglog(channels[:,0],channels[:,1], 'o' ) plt.xlabel('CH0') plt.ylabel('CH1') a2 = plt.subplot(222) plt.loglog(channels[:,2],channels[:,3], 'o' ) plt.xlabel('CH2') plt.ylabel('CH3') |
channels is a 4xN array of floats, where N is the number of events. Each column corresponds to one out of four Ipimb channels. |
]]></ac:plain-text-body></ac:structured-macro> |
test |
test |
Test |
|
array of limits from graphical input |
array of limits from graphical input |
|
|
axes(a1) hold on lims(1:2,:) = ginput(2); axes(a2) hold on lims(3:4,:) = ginput(2); |
lims = np.zeros((4,2),dtype="float") plt.axes(a1) plt.hold(True) lims[0:2,:] = plt.ginput(2) plt.axes(a2) plt.hold(True) lims[2:4,:] = plt.ginput(2) |
In MatLab, |
|
|
|
|
|
filter |
filter |
|
|
fbool1 = (channels(:,1)>min(lims(1:2,1)))&(channels(:,1)<max(lims(1:2,1))) fbool2 = (channels(:,2)>min(lims(1:2,2)))&(channels(:,2)<max(lims(1:2,2))); fbool = fbool1&fbool2 loglog(channels(fbool,1),channels(fbool,2),'or') fbool3 = (channels(:,3)>min(lims(3:4,3)))&(channels(:,3)<max(lims(3:4,3))) fbool4 = (channels(:,4)>min(lims(3:4,4)))&(channels(:,4)<max(lims(3:4,4))); fbool = fbool3&fbool4 loglog(channels(fbool,3),channels(fbool,4),'or') |
fbools0 = (channels[:,0]>lims[:,0].min())&(channels[:,0]<lims[:,0].max()) fbools1 = (channels[:,1]>lims[:,1].min())&(channels[:,1]<lims[:,1].max()) fbools = fbools0 & fbools1 fbools2 = (channels[:,2]>lims[:,2].min())&(channels[:,2]<lims[:,2].max()) fbools3 = (channels[:,3]>lims[:,3].min())&(channels[:,3]<lims[:,3].max()) fbools = fbools2&fbools3 |
Comment |
|
|
|
|