Above: A screenshot of realtime AMI plots generated in the TMO hutch during the LW08 experiment

Setting up Environment

NOTE: if you want to run ami on LCLS1 data, you can do it by generating an h5 file with the LCLS1 MPIDataSource.

To get the ami environment on the psana cluster:

ssh psana
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh

Run the appropriate command from the "examples" sections below and do (for example) the following:

• click on "apply" (menu item near the top left) to activate the analysis "graph"
• click on one of the green boxes to open a plot

For examples that load a predefined graph (.fc file, for "flow chart") you should see a window like this, while examples with no .fc file will have an empty area in the upper right:

How To Learn AMI

Seshu Yamajala (who along with Dan Damiani is one of the two main authors of AMI) has an 8-minute getting-started demonstration video:

ami_tutorial.mp4

After viewing that video, you can:

• Start with the Simple Introductory Examples which run from offline files
• Try adding red data-source boxes by dragging names from the top left area into the main grey graph area
• Try adding white/blue operation boxes and green display boxes by dragging names from the lower left area into the main grey graph area.  Operation boxes that are white do not "gather" data from multiple nodes, operation boxes that are blue do.
• Connect boxes by dragging outputs of one box to the inputs of another box.  Be aware that ami only lets you connect inputs/outputs of the same data-type (hover mouse over the box inputs/outputs to see the types in the lower-right text area)
• Right-click on boxes to see different options that are available (e.g. adding more inputs to some boxes, removing boxes)
• Try a 1D or 2D ROI box that allows you to specify ROI's graphically
• Learn how to search for data-sources (upper left names) and operations (lower left names) using the two search boxes
• Learn how to use a PythonEditor box to be able to add your own python code to AMI
• Learn how to manipulate the pyqtgraph display plots that AMI uses.  Many things can be adjusted in these plots (e.g. zooming, color scale, ...).  The small "wrench" icon on the top left of each plot can also be useful for customizing (e.g. line colors, auto-ranging)
• Develop graphs in advance of your experiment using offline data.  There is a reasonable chance that data exists that would be sufficient for you to do this development in advance (send email to pcds-ana-l@slac.stanford.edu if you want help identifying appropriate data for your use-case)
• Also email pcds-ana-l@slac.stanford.edu if you think there are commonly-useful features that should be added

Simple Introductory Examples

NOTE: when learning, or developing new graphs it can be useful to slow ami down to 1Hz, using the "-b" and "-f" options shown in the first example

Viewing Images

• Login to a psana node and setup the environment as shown above, then type:

ami-local -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/cds/data/psdm/prj/public01/xtc

• When the AMI window comes up in the "search sources" box on the top left type "opal".  That will select only detectors that have the string "opal" in them
• drag the text "tmo_atmopal:raw:image" into the rightmost part of the ami window, a red box will appear
• click on the red box. a Pyqtgraph window should appear with the image, which will update at 1Hz.  The plot has many user-settable attributes.  You can try clicking the "wrench" icon in the upper left and removing the Auto-Range and Auto-Levels settings (followed by "Apply"): this will give you more control of the color scale using the horizontal yellow control bars on the right.

Step-Scan Example

A 2D-ROI, summed over pixels.  Mean value of pixel-sum is plotted vs. step_value.

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/epixroi_meanvscan.fc psana://exp=rixx45619,run=122,dir=/cds/data/psdm/prj/public01/xtc/

TMO Examples

Example includes:

• waveform viewing, ROI, integration and normalization of peak area
• waveform peak finding, histogramming, and scaling of axes using a polynomial "calibration"
  
  

POP example

 ami-local -l /reg/g/psdm/tutorials/ami2/tmo/pop.fc

XTCAV example

ami-local -l /reg/g/psdm/tutorials/ami2/tmo/amox23616_run137.fc

HSD example

ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hsd.fc -f repeat=true psana://exp=tstx00417,run=82

Blobfinder example

ami-local -f repeat=true -f interval=0.1 -f init=1 -l /reg/g/psdm/tutorials/ami2/tmo/blobfinder.fc psana://exp=tmoc00518,run=1

HitFinder example

ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hitfinder.fc

RIX Examples

FlyScan: Mean Vs. Scan Value

An example of using the MeanVsScan Box to plot an average parameter (in this case the width of a distribution) vs. the fly-scanned PV.  "fly scan" means the scan value is changed independently while the DAQ free-runs.  Compared to a "step scan" (see below) it has the disadvantage that there is no precise knowledge of the scan variable value at each event, since the scan variable is moving during data taking and is only recorded by the DAQ at ~1Hz.  Note that this example uses a "MeanVsScan" box which can be used for both fly-scans and step-scans: for fly scans the "binned" option (and histogram limits) of the MeanVsScan box should be selected in advance.

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_flyscan.fc psana://exp=rixx43518,run=34,dir=/cds/data/psdm/prj/public01/xtc

Waveform Sideband Subtraction

Demonstrates a 2D ROI, projection, and subtraction of average of 1D ROI of the projection.

ami-local -b 1 -f interval=1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_bkgdsub.fc psana://exp=rixx43518,run=34,dir=/cds/data/psdm/prj/public01/xtc

Step-Scan With Gaussian Fit

A "step scan" is a mode where the DAQ is paused whenever the scan variable changes.  Unlike a fly-scan, it is possible to assigned each event to a well-defined scan variable value.  These scans are typically controlled by the hutch-python tool, which uses the bluesky tool from BNL.  In this particular example, pedestal subtraction helps the fit significantly and is done in a kludgy way with using /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.py which generates a numpy pedestal file /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.npy.  The pedestal subtraction is done with a PythonEditor box.

ami-local -f interval=0.1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_width_scan.fc psana://exp=rixx43518,run=45,dir=/cds/data/psdm/prj/public01/xtc

Photon Finding and Check of Linearity

Example includes:

• photon centroid finding on vertically binned Andor camera
• linearity check of normalization (andor camera intensity vs. acqiris waveform intensity) with pearson-R calculation as a function of time

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/run22.fc hdf5:///reg/g/psdm/tutorials/ami2/neh2.2/worker.json

Waveform Integration vs. Delay Stage

Example includes:

• Plot of integration of acqiris vs delay stage
  
  

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run37.fc

Normalization of Andor

Example includes:

• 1d ROI of Andor
• Normalization of ROI by integration of acqiris
  
  

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run23.fc

Ratio of Peak Areas and EPICS Export of Result

Example includes:

• Ratio of peak area and export over EPICS
  
  

ami-local -e ami -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlq7615_run176.fc
pvlist localhost:5075
pvget ami:ana:graph:data:peak_area

XPP Example

 ami-local hdf5:///reg/g/psdm/tutorials/ami2/xpp/xpp.json

Random Data Examples

Calculator example

Example includes:

• calculator executing based on conditional logic
  
  

 ami-local -l /reg/g/psdm/tutorials/ami2/random/calculator.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Complex example

 Example includes:

• binning based on conditional logic

ami-local -l /reg/g/psdm/tutorials/ami2/random/complex_example.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Timestamp Plot example

Example includes:

• Plotting average of integration of waveform against time

ami-local -l /reg/g/psdm/tutorials/ami2/random/timeplot.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Export Example

Example includes:

• Averaging waveform over 100 events and exporting result from global collector to workers
• Background subtraction of averaged waveform on workers

ami-local -e test -l /reg/g/psdm/tutorials/ami2/random/export.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Event Filtering in AMI

Often an analysis wants to treat events differently based upon something in the data.  A common example is the presence of an eventcode, which may indicate the firing of a pump laser or some other condition.  If the selection logic is put into a PythonEditor box, then the editor box should return python's None object to indicate the event flow is to be terminated here; returning a value (even 0) causes the flow to continue with that value and possibly weighting your statistics in unexpected ways.  An example flowchart and PythonEditor box is shown below.  Other boxes can also handle filtering with their "conditional" attributes.

Loading Data in AMI

There are three different ways to load data in AMI, either through using the gui, providing a json file, or by passing command line arguments. The different methods are described below.

Command Line Configuration

Given a psana experiment and run number the xtc2 files can be loaded using the following syntax:

ami-local psana://exp=tstx00417,run=82

A more complex example that loads a saved flowchart, sets the "heartbeat" interval (-b) to 1 (defaults to 10) and fetches events from disk at 1Hz ("-f interval", defaults to as fast as possible) is:

ami-local -l spike.fc -b 1 -f interval=1 psana://exp=rixdaq18,run=18

The "-f init_time=NN" (where "NN" is a number of seconds) can also be important.  It tells the data reading to wait and give the analysis side of ami a chance to startup.  This is necessary because the data reading portion starts immediately and can drop events, which is what ami needs to do when running in realtime (to ensure that it is only looking at recent data).

Loading Data through the GUI

The first option is to run ami-local without any flags. Once the main window appears, click the Configure button and the window in the image above will appear. From there hdf5 files or xtc2 files can be loaded by clicking on Add. Only one type of file may be loaded at a time, either hdf5 or xtc2, but multiple files can be loaded. AMI will loop through all the events in each file in the order they are loaded. The different options are explained below.

Note: If a source is configured through the gui configuration window, the source configuration will be saved with the graph and can be automatically reloaded by running ami-local -l <graph.fc>

JSON Configuration File

 {
    "interval": 0.01,
    "init_time": 0.5,
    "repeat": true,
	"files": ["run22.h5"]
}

Additionally a JSON file containing a source configuration may be provided. A URI to the file where the prefix contains the source type must be passed to AMI. For example to load the above file run ami-local hdf5:///worker.json to use the psana source run ami-local psana:///worker.json. 

Configuration Flags

These flags can be passed to the source using any of the three methods above, the gui configuration window, a json file, or using the -f command line flag. To specify multiple flags using the command line pass -f multiple times.  For example:

ami-local -f interval=1 psana://exp=tmoc00118,run=4,dir=/ffb01/data/tmo/tmoc00118/xtc/

Reporting AMI Issues

Users are encouraged to submit AMI bug reports, feature requests, and questions through github issues: https://github.com/slac-lcls/ami/issues

If appropriate, please include an experiment, run number, and example graph so AMI can be run in off-line mode to reproduce issues. 

Screenshots

From the first user experiment of 2020: