Interactive data analysis with iPython

Here are some examples of how you can use interactive python to analyze your data. We use matplotlib to draw plots from numpy arrays. The numpy arrays can be read directly from an HDF5 file, or can be extracted from XTC files via pyana/psana. Please note that currently there is no way to run pyana from IPython, but you can run a pyana job and launch ipython at the end to play with the plots/arrays. For an example of how to set up the latter, see XtcExplorer#IPython

• 'ipython' (http://ipython.org/^{Image Added}) is an enhanced python shell for interactive use. Many of the examples here would work equally well with a 'regular' python shell.
• plotting is done with 'matplotlib' (http://matplotlib.sourceforge.net/^{Image Added})

Interactive access to XTC data

An interactive framework based on ipython is currently being explored, but does not exist yet.

You have the option of working with HDF5 files in an interactive ipython session. Be aware that when you work with HDF5 files, arrays from different sources (detectors) may not be synchronized. You will need to time-order and/or synchronize them yourself if you want to correlate data from different sources! See e.g. How to access HDF5 data from Python for how to do this.

These examples are mostly python rewrites of matlab functions provided by XPP. Thanks to H. Lemke for matlab examples and advice.

The components

• ipython
  ... an interactive python shell
  External links:
  • http://ipython.github.com/ipython-doc/dev/interactive/index.html
  • http://ipython.scipy.org/moin/Cookbook
  • http://pages.physics.cornell.edu/~myers/teaching/ComputationalMethods/python/ipython.html

• matplotlib pyplot / pylab
  ... an open-source plotting tool

...

Benefits of XTC files is that they are available immediately, and you can start analyzing before the run is done collecting. The benefits of using the LCLS framework(s) is that each event is easily extracted and you don't have to worry about time-ordering or synchronizing data from different devices.

If you'd like to use XTC files, the options that exist are:

• ipython in combination with pyana. The XTC Explorer gives you this option (XtcExplorer#IPython). Note that currently there is no way to run pyana from IPython, but you can run a pyana job and launch ipython at the end to play with the plots/arrays.

Examples

These examples are mostly python rewrites of matlab functions provided by XPP. Thanks to H. Lemke for matlab examples and advice.

...

The library

• pymatlab.py
  ... a module implementing in python some of the tools written by Henrik/XPP for matlab. For those familiar with the XPP matlab tools, the functions here should be intuitive to use. Only a few functions have been implemented thus far... (feel free to contribute).
  

  Code Block
  none none title Starting iPython borderStyle solid
  [ofte@psana0XXX myrelease]$ ipython Python 2.4.3 (#1, Nov 3 2010, 12:52:40) Type "copyright", "credits" or "license" for more information. IPython 0.9.1 -- An enhanced Interactive Python. ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object'. ?object also works, ?? prints more.

  
  Loading the library. Normally 'import pymatlab' would be recommended, but if you do 'from pymatlab import *', all the functions defined in this module gets loaded in the current namespace, and you can see them in your workspace. This might be easier for interactive work.

  Code Block
  none none borderStyle solid
  In [1]: from pymatlab import * Pretend this is matlab

  
  who gives you a short list of workspace contents

  Code Block
  none none borderStyle solid
  In [2]: who H5getobjnames ScanInput ScanOutput filtvec findmovingmotor getSTDMEANfrac_from_startpoint get_filter get_limits get_limits_automatic get_limits_channelhist get_limits_correlation get_limits_corrfrac h5py np plt rdXPPdata runexpNO2fina scan scaninput

  
  whos gives you a longer list of workspace contents

  Code Block
  none none borderStyle solid
  In [3]: whos Variable Type Data/Info -------------------------------------------------------- H5getobjnames function <function H5getobjnames at 0x2b57de8> ScanInput type <class 'pymatlab.ScanInput'> ScanOutput type <class 'pymatlab.ScanOutput'> filtvec function <function filtvec at 0x2b57f50> findmovingmotor function <function findmovingmotor at 0x2b57d70> getSTDMEANfrac_from_startpoint function <function getSTDMEANfrac_<...>_startpoint at 0x2b581b8> get_filter function <function get_filter at 0x2b57ed8> get_limits function <function get_limits at 0x2b58050> get_limits_automatic function <function get_limits_automatic at 0x2b58230> get_limits_channelhist function <function get_limits_channelhist at 0x2b582a8> get_limits_correlation function <function get_limits_correlation at 0x2b580c8> get_limits_corrfrac function <function get_limits_corrfrac at 0x2b58140> h5py module <module 'h5py' from '/reg<...>ython/h5py/__init__.pyc'> np module <module 'numpy' from '/re<...>thon/numpy/__init__.pyc'> plt module <module 'matplotlib.pyplo<...>n/matplotlib/pyplot.pyc'> rdXPPdata function <function rdXPPdata at 0x2b57c80> runexpNO2fina function <function runexpNO2fina at 0x2b57e60> scan ScanOutput <pymatlab.ScanOutput object at 0x2b60536bee90> scaninput ScanInput <pymatlab.ScanInput object at 0x2b60536b4e90>

Examples

1) Select limits from graphical input and plot filtered IPIMB

...

%
%
%
a1 = subplot(121);
loglog(channels(:,1),channels(:,2),'o')
xlabel('CH0')
ylabel('CH1')
a2 = subplot(122);
loglog(channels(:,3),channels(:,4),'o')
xlabel('CH2')
ylabel('CH3')

import matplotlib.pyplot as plt
import numpy as np

a1 = plt.subplot(221)
plt.loglog(channels[:,0],channels[:,1], 'o' )
plt.xlabel('CH0')
plt.ylabel('CH1')
a2 = plt.subplot(222)
plt.loglog(channels[:,2],channels[:,3], 'o' )
plt.xlabel('CH2')
plt.ylabel('CH3')

axes(a1)
hold on
lims(1:2,:) = ginput(2);

axes(a2)
hold on
lims(3:4,:) = ginput(2);

lims = np.zeros((4,2),dtype="float")

plt.axes(a1)
plt.hold(True)
lims[0:2,:] = plt.ginput(2)

plt.axes(a2)
plt.hold(True)
lims[2:4,:] = plt.ginput(2)

fbool1 = (channels(:,1)>min(lims(1:2,1)))&(channels(:,1)<max(lims(1:2,1)))
fbool2 = (channels(:,2)>min(lims(1:2,2)))&(channels(:,2)<max(lims(1:2,2)));
fbool = fbool1&fbool2
loglog(channels(fbool,1),channels(fbool,2),'or')

fbool3 = (channels(:,3)>min(lims(3:4,3)))&(channels(:,3)<max(lims(3:4,3)))
fbool4 = (channels(:,4)>min(lims(3:4,4)))&(channels(:,4)<max(lims(3:4,4)));
fbool = fbool3&fbool4
loglog(channels(fbool,3),channels(fbool,4),'or') 

fbools0 = (channels[:,0]>lims[:,0].min())&(channels[:,0]<lims[:,0].max())
fbools1 = (channels[:,1]>lims[:,1].min())&(channels[:,1]<lims[:,1].max())
fbools = fbools0 & fbools1

fbools2 = (channels[:,2]>lims[:,2].min())&(channels[:,2]<lims[:,2].max())
fbools3 = (channels[:,3]>lims[:,3].min())&(channels[:,3]<lims[:,3].max())
fbools = fbools2&fbools3