1. About this tutorial

The main objective of this session is to introduce and to explain the new Python interface for accessing LCLS data from analysis applications. The new software framework is known as the "Interactive psana" or just ipsana. The first idea of implementing such tool was suggested around 1.5 years ago at the joint PCDS/SRD meeting (look for ipsana). Though its underlying machinery is largely based on the batch version of psana the interface to the data is much simple, more intuitive, and it requires much less code to be written by a user in order to get to "that CSPad image" (EPICS PV, etc.). The new framework won't work for everyone, especially for those users who have either heavily invested into the modular code of the batch frameworks, or who needs the performance of modules written in C++. Still our intent is to demonstrate the power of the new approach and to encourage using the tool where it seems to be appropriate.

What is beyond a scope of this tutorial

• This isn't an explanation how to do the data analysis. Note that our goal is to explain basic techniques for getting to your data, not for using it!
• This is not a Users or Reference Guide for the interactive framework

The pyana users, attention!

As it's been announced earlier, the pyana framework will be phased out at some point. There is a variety of reasons why:

...

The last two features are opening an interesting possibility of using psana for real-time monitoring of data while benefiting from reusing
the same code which might be developed for the traditional OFFLINE processing/analysis.

2. Things to know before to run the examples

The location of examples

We put all examples for today's session at the following diretory:

/reg/g/psdm/tutorials

Data files used in the examples

In order to make our examples as close to the "real" analysis environment as possible we chose to create 6 pseudo experiments (one per instrument):

...

All data files are open for reading by anyone who can log onto PCDS computers. Moreover, those directories (like scratch/, ftc/) are open for writing by anyone. And yes, one can also see these experiments in the Web Portal.

Setting up your environment

• make sure you can run X11 applications. Most examples of this tutorial will do a simple visualization.
• log onto any machine of interactive analysis clusters psananeh or psanafeh
• make sure you sources (just once) one of the following scripts (depending on which UNIX shell you are using):

  Code Block
  . /reg/g/psdm/etc/sit_env.sh source /reg/g/psdm/etc/sit_env.csh

• run (just once) the following command which will set up a proper OFFLINE Analysis environment for the latest analysis release:

  Code Block
  sit_setup ana-current

...

psana
[error:2013-06-06 20:54:44.131:PSAnaApp.cpp:218] no analysis modules specified

3. Basic examples

This section presents a few simple scripts which have been developed to underline the main ideas behind the framework's API. The code of the examples along with a simple HOWTO file can be found at:

/reg/g/psdm/tutorials/common/data_access_methods/

Printing identifiers of all events of a run

First try this:

./print_event_id.py

...

./print_event_id.py
     1: XtcEventId(run=366, time=2013-04-21 04:37:39.343773772-07, fiducials=38877, ticks=329342, vector=19553)
     2: XtcEventId(run=366, time=2013-04-21 04:37:39.360457259-07, fiducials=38883, ticks=331442, vector=19554)
     3: XtcEventId(run=366, time=2013-04-21 04:37:39.377123777-07, fiducials=38889, ticks=330560, vector=19555)
     4: XtcEventId(run=366, time=2013-04-21 04:37:39.393797466-07, fiducials=38895, ticks=329762, vector=19556)
     5: XtcEventId(run=366, time=2013-04-21 04:37:39.410477971-07, fiducials=38901, ticks=331204, vector=19557)
     6: XtcEventId(run=366, time=2013-04-21 04:37:39.427145705-07, fiducials=38907, ticks=331036, vector=19558)
     7: XtcEventId(run=366, time=2013-04-21 04:37:39.443816588-07, fiducials=38913, ticks=329370, vector=19559)
     8: XtcEventId(run=366, time=2013-04-21 04:37:39.460499778-07, fiducials=38919, ticks=331414, vector=19560)
     9: XtcEventId(run=366, time=2013-04-21 04:37:39.477167658-07, fiducials=38925, ticks=330616, vector=19561)
    10: XtcEventId(run=366, time=2013-04-21 04:37:39.493840079-07, fiducials=38931, ticks=329720, vector=19562)
    ...

Printing a catalog of event components

The sample can be run like this:

...

Note that event components obtained through this API will be objects of various classes. A full catalog of those objects can be found in the DOXYGEN documentation which is auto-generated from the code of the OFFLINE releases.

Iterating over scans and events

Some of our experiments (in particular XPP) are heavily relying on so called scans (also known as "Calibration Transitions*) while taking their data. Each DAQ run has one or many scans. Events are recorded in a scope of a scan. The new framework has a special provision for scans through the iterator of scans. The idea begin the following example is:

...

They both do the same. The only subtle difference is which data format they're suing. The first example will read XTC files, while the second one will read HDF5 files. When running these examples you should notice differences in their performance. They're explained by different organization of data in XTC vs HDF5 formats. We'll be happy to provide you with an explanation if you'll be interested in it.

4. Instrument-specific examples

This section includes a number of examples which are relevant for different instruments. Their primary meaning is to illustrate how to access data objects which are

XCS: movie

The code of examples is found at:

/reg/g/psdm/tutorials/xcs/princeton_movie/

SXR: correlation plots for signals from GDM and Diode

The code of examples is found at:

/reg/g/psdm/tutorials/sxr/gmd_vs_diode/

CXI: diffraction patterns on the CSPad detector

The code of examples is found at:

...

./dump_2x1_elements.py
./frame_reco.py
./frame_reco_calib.py

5. Doing something less trivial

Custom HDF5 translator written by a user

A problem:

• Let's suppose we need to write a data extraction tool to extract images (CSPad, Princeton, etc) from XTC files and make then available for further analysis in Matlab. At this point we should already know how to get images from the raw files using ipsana. Now the only remaining problem is to store them in some form which may be readable from Matlab (assuming we're looking at some reasonable performance).

...