Above: A screenshot of realtime AMI plots generated in the TMO hutch during the LW08 experiment
Setting up Environment
NOTE: if you want to run ami on LCLS1 data, you can do it by generating an h5 file with the LCLS1 MPIDataSource.
To get the ami environment on the psana cluster:
ssh psana source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
Run the appropriate command from the "examples" sections below and do (for example) the following:
- click on "apply" (menu item near the top left) to activate the analysis "graph"
- click on one of the green boxes to open a plot
For examples that load a predefined graph (.fc file, for "flow chart") you should see a window like this, while examples with no .fc file will have an empty area in the upper right:
How To Learn AMI
Seshu Yamajala (who along with Dan Damiani is one of the two main authors of AMI) has an 8-minute getting-started demonstration video:
After viewing that video, you can:
- Start with the Simple Introductory Examples which run from offline files
- Try adding red data-source boxes by dragging names from the top left area into the main grey graph area
- Try adding white/blue operation boxes and green display boxes by dragging names from the lower left area into the main grey graph area. Operation boxes that are white do not "gather" data from multiple nodes, operation boxes that are blue do.
- Connect boxes by dragging outputs of one box to the inputs of another box. Be aware that ami only lets you connect inputs/outputs of the same data-type (hover mouse over the box inputs/outputs to see the types in the lower-right text area)
- Right-click on boxes to see different options that are available (e.g. adding more inputs to some boxes, removing boxes)
- Try a 1D or 2D ROI box that allows you to specify ROI's graphically
- Learn how to search for data-sources (upper left names) and operations (lower left names) using the two search boxes
- Learn how to use a PythonEditor box to be able to add your own python code to AMI
- Learn how to manipulate the pyqtgraph display plots that AMI uses. Many things can be adjusted in these plots (e.g. zooming, color scale, ...). The small "wrench" icon on the top left of each plot can also be useful for customizing (e.g. line colors, auto-ranging)
- Develop graphs in advance of your experiment using offline data. There is a reasonable chance that data exists that would be sufficient for you to do this development in advance (send email to pcds-ana-l@slac.stanford.edu if you want help identifying appropriate data for your use-case)
- Also email pcds-ana-l@slac.stanford.edu if you think there are commonly-useful features that should be added
Simple Introductory Examples
NOTE: when learning, or developing new graphs it can be useful to slow ami down to 1Hz, using the "-b" and "-f" options shown in the first example
Viewing Images
- Login to a psana node and setup the environment as shown above, then type:
ami-local -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/cds/data/psdm/prj/public01/xtc
- When the AMI window comes up in the "search sources" box on the top left type "opal". That will select only detectors that have the string "opal" in them
- drag the text "tmo_atmopal:raw:image" into the rightmost part of the ami window, a red box will appear
- click on the red box. a Pyqtgraph window should appear with the image, which will update at 1Hz. The plot has many user-settable attributes. You can try clicking the "wrench" icon in the upper left and removing the Auto-Range and Auto-Levels settings (followed by "Apply"): this will give you more control of the color scale using the horizontal yellow control bars on the right.
Step-Scan Example
A 2D-ROI, summed over pixels. Mean value of pixel-sum is plotted vs. step_value.
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/epixroi_meanvscan.fc psana://exp=rixx45619,run=122,dir=/cds/data/psdm/prj/public01/xtc/
TMO Examples
Example includes:
- waveform viewing, ROI, integration and normalization of peak area
- waveform peak finding, histogramming, and scaling of axes using a polynomial "calibration"
POP example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/pop.fc
XTCAV example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/amox23616_run137.fc
HSD example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hsd.fc -f repeat=true psana://exp=tstx00417,run=82
Blobfinder example
ami-local -f repeat=true -f interval=0.1 -f init=1 -l /reg/g/psdm/tutorials/ami2/tmo/blobfinder.fc psana://exp=tmoc00518,run=1
HitFinder example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hitfinder.fc
RIX Examples
FlyScan: Mean Vs. Scan Value
An example of using the MeanVsScan Box to plot an average parameter (in this case the width of a distribution) vs. the fly-scanned PV. "fly scan" means the scan value is changed independently while the DAQ free-runs. Compared to a "step scan" (see below) it has the disadvantage that there is no precise knowledge of the scan variable value at each event, since the scan variable is moving during data taking and is only recorded by the DAQ at ~1Hz. Note that this example uses a "MeanVsScan" box which can be used for both fly-scans and step-scans: for fly scans the "binned" option (and histogram limits) of the MeanVsScan box should be selected in advance.
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_flyscan.fc psana://exp=rixx43518,run=34,dir=/cds/data/psdm/prj/public01/xtc
Waveform Sideband Subtraction
Demonstrates a 2D ROI, projection, and subtraction of average of 1D ROI of the projection.
ami-local -b 1 -f interval=1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_bkgdsub.fc psana://exp=rixx43518,run=34,dir=/cds/data/psdm/prj/public01/xtc
Step-Scan With Gaussian Fit
A "step scan" is a mode where the DAQ is paused whenever the scan variable changes. Unlike a fly-scan, it is possible to assigned each event to a well-defined scan variable value. These scans are typically controlled by the hutch-python tool, which uses the bluesky tool from BNL. In this particular example, pedestal subtraction helps the fit significantly and is done in a kludgy way with using /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.py which generates a numpy pedestal file /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.npy. The pedestal subtraction is done with a PythonEditor box.
ami-local -f interval=0.1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_width_scan.fc psana://exp=rixx43518,run=45,dir=/cds/data/psdm/prj/public01/xtc
Photon Finding and Check of Linearity
Example includes:
- photon centroid finding on vertically binned Andor camera
- linearity check of normalization (andor camera intensity vs. acqiris waveform intensity) with pearson-R calculation as a function of time
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/run22.fc hdf5:///reg/g/psdm/tutorials/ami2/neh2.2/worker.json
Waveform Integration vs. Delay Stage
Example includes:
- Plot of integration of acqiris vs delay stage
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run37.fc
Normalization of Andor
Example includes:
- 1d ROI of Andor
- Normalization of ROI by integration of acqiris
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run23.fc
Ratio of Peak Areas and EPICS Export of Result
Example includes:
- Ratio of peak area and export over EPICS
ami-local -e ami -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlq7615_run176.fc pvlist localhost:5075 pvget ami:ana:graph:data:peak_area
XPP Example
ami-local hdf5:///reg/g/psdm/tutorials/ami2/xpp/xpp.json
Random Data Examples
Calculator example
Example includes:
- calculator executing based on conditional logic
ami-local -l /reg/g/psdm/tutorials/ami2/random/calculator.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Complex example
Example includes:
- binning based on conditional logic
ami-local -l /reg/g/psdm/tutorials/ami2/random/complex_example.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Timestamp Plot example
Example includes:
- Plotting average of integration of waveform against time
ami-local -l /reg/g/psdm/tutorials/ami2/random/timeplot.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Export Example
Example includes:
- Averaging waveform over 100 events and exporting result from global collector to workers
- Background subtraction of averaged waveform on workers
ami-local -e test -l /reg/g/psdm/tutorials/ami2/random/export.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Reference Plot Example
In AMI one can export the x,y values of a waveform to a .csv file using the "camera" icon shown in the top left of the above window. That .csv file can then be loaded with a LoadReference1D box which makes those (static) values available for later analysis/display. NOTE: if you overlay the reference data with a signal in a plot be careful that the horizontal scales for both are the same.
ami-local -l /reg/g/psdm/tutorials/ami2/random/ref_waveform.fc -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/cds/data/psdm/prj/public01/xtc
Event Filtering in AMI
Often an analysis wants to treat events differently based upon something in the data. A common example is the presence of an eventcode, which may indicate the firing of a pump laser or some other condition. If the selection logic is put into a PythonEditor box, then the editor box should return python's None object to indicate the event flow is to be terminated here; returning a value (even 0) causes the flow to continue with that value and possibly weighting your statistics in unexpected ways. An example flowchart and PythonEditor box is shown below. Other boxes can also handle filtering with their "conditional" attributes.
Loading Data in AMI
There are three different ways to load data in AMI, either through using the gui, providing a json file, or by passing command line arguments. The different methods are described below.
Command Line Configuration
Given a psana experiment and run number the xtc2 files can be loaded using the following syntax:
ami-local psana://exp=tstx00417,run=82
A more complex example that loads a saved flowchart, sets the "heartbeat" interval (-b) to 1 (defaults to 10) and fetches events from disk at 1Hz ("-f interval", defaults to as fast as possible) is:
ami-local -l spike.fc -b 1 -f interval=1 psana://exp=rixdaq18,run=18
The "-f init_time=NN" (where "NN" is a number of seconds) can also be important. It tells the data reading to wait and give the analysis side of ami a chance to startup. This is necessary because the data reading portion starts immediately and can drop events, which is what ami needs to do when running in realtime (to ensure that it is only looking at recent data).
Loading Data through the GUI
The first option is to run ami-local without any flags. Once the main window appears, click the Configure button and the window in the image above will appear. From there hdf5 files or xtc2 files can be loaded by clicking on Add. Only one type of file may be loaded at a time, either hdf5 or xtc2, but multiple files can be loaded. AMI will loop through all the events in each file in the order they are loaded. The different options are explained below.
Note: If a source is configured through the gui configuration window, the source configuration will be saved with the graph and can be automatically reloaded by running ami-local -l <graph.fc>
JSON Configuration File
{ "interval": 0.01, "init_time": 0.5, "repeat": true, "files": ["run22.h5"] }
Additionally a JSON file containing a source configuration may be provided. A URI to the file where the prefix contains the source type must be passed to AMI. For example to load the above file run ami-local hdf5:///worker.json to use the psana source run ami-local psana:///worker.json.
Configuration Flags
Flag | Description |
---|---|
interval | Time in seconds to sleep between events |
init_time | Time in seconds to sleep before emitting the first event |
repeat | Loop over files/events after reaching end |
files | Comma separated list of files to load (either xtc2 or hdf5) |
These flags can be passed to the source using any of the three methods above, the gui configuration window, a json file, or using the -f command line flag. To specify multiple flags using the command line pass -f multiple times. For example:
ami-local -f interval=1 psana://exp=tmoc00118,run=4,dir=/ffb01/data/tmo/tmoc00118/xtc/
Reporting AMI Issues
Users are encouraged to submit AMI bug reports, feature requests, and questions through github issues: https://github.com/slac-lcls/ami/issues
If appropriate, please include an experiment, run number, and example graph so AMI can be run in off-line mode to reproduce issues.
Screenshots
From the first user experiment of 2020: