Above: A screenshot of realtime AMI plots generated in the TMO hutch during the LW08 experiment
Setting up Environment
NOTE: if you want to run ami on LCLS1 data, you can do it by generating an h5 file with the LCLS1 MPIDataSource.
To get the ami environment on the psana cluster:
ssh psana source /sdf/group/lcls/ds/ana/sw/conda2/manage/bin/psconda.sh
Run the appropriate command from the "examples" sections below and do (for example) the following:
- click on "apply" (menu item near the top left) to activate the analysis "graph"
- click on one of the green boxes to open a plot
For examples that load a predefined graph (.fc file, for "flow chart") you should see a window like this, while examples with no .fc file will have an empty area in the upper right:
How To Learn AMI
Seshu Yamajala (who along with Dan Damiani is one of the two main authors of AMI) has an 8-minute getting-started demonstration video:
After viewing that video, you can:
- Start with the Simple Introductory Examples which run from offline files
- Try adding red data-source boxes by dragging names from the top left area into the main grey graph area
- Try adding white/blue operation boxes and green display boxes by dragging names from the lower left area into the main grey graph area. Operation boxes that are white do not "gather" data from multiple nodes, operation boxes that are blue do.
- Connect boxes by dragging outputs of one box to the inputs of another box. Be aware that ami only lets you connect inputs/outputs of the same data-type (hover mouse over the box inputs/outputs to see the types in the lower-right text area)
- Right-click on boxes to see different options that are available (e.g. adding more inputs to some boxes, removing boxes)
- Try a 1D or 2D ROI box that allows you to specify ROI's graphically
- Learn how to search for data-sources (upper left names) and operations (lower left names) using the two search boxes
- Learn how to use a PythonEditor box to be able to add your own python code to AMI
- Learn how to manipulate the pyqtgraph display plots that AMI uses. Many things can be adjusted in these plots (e.g. zooming, color scale, ...). The small "wrench" icon on the top left of each plot can also be useful for customizing (e.g. line colors, auto-ranging)
- Develop graphs in advance of your experiment using offline data. There is a reasonable chance that data exists that would be sufficient for you to do this development in advance (send email to pcds-ana-l@slac.stanford.edu if you want help identifying appropriate data for your use-case)
- Also email pcds-ana-l@slac.stanford.edu if you think there are commonly-useful features that should be added
Simple Introductory Examples
NOTE: when learning, or developing new graphs it can be useful to slow ami down to 1Hz, using the "-b" and "-f" options shown in the first example. "-b" is a "heartBeat" interval in milliseconds (using the timestamps of the events being read!) and controls how often the plots update. "-f interval=1" controls the interval (in seconds) between events read from disk.
Viewing Images
- Login to a psana node and setup the environment as shown above, then type:
ami-local -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/sdf/data/lcls/ds/prj/public01/xtc/
- When the AMI window comes up in the "search sources" box on the top left type "opal". That will select only detectors that have the string "opal" in them
- drag the text "tmo_atmopal:raw:image" into the rightmost part of the ami window, a red box will appear
- click on the red box. a Pyqtgraph window should appear with the image, which will update at 1Hz. The plot has many user-settable attributes. You can try clicking the "wrench" icon in the upper left and removing the Auto-Range and Auto-Levels settings (followed by "Apply"): this will give you more control of the color scale using the horizontal yellow control bars on the right.
Step-Scan Example
A 2D-ROI, summed over pixels. Mean value of pixel-sum is plotted vs. step_value. The "-b" and "-f" flags slow down the data so it doesn't go by too quickly (there are unfortunately some race conditions in ami between the start of the analysis graph and the reading of the data).
ami-local -b 100 -f interval=0.002 -l /reg/g/psdm/tutorials/ami2/neh2.2/epixroi_meanvscan.fc psana://exp=rixx45619,run=122,dir=/sdf/data/lcls/ds/prj/public01/xtc//
Image reconstruction using geometry
Mask and RoiArch examples
TMO Examples
Example includes:
- waveform viewing, ROI, integration and normalization of peak area
- waveform peak finding, histogramming, and scaling of axes using a polynomial "calibration"
POP example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/pop.fc
XTCAV example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/amox23616_run137.fc
HSD example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hsd.fc -f repeat=true psana://exp=tstx00417,run=82
Blobfinder example
ami-local -f repeat=true -f interval=0.1 -f init=1 -l /reg/g/psdm/tutorials/ami2/tmo/blobfinder.fc psana://exp=tmoc00518,run=1,dir=/sdf/data/lcls/ds/prj/public01/xtc//
HitFinder example
ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hitfinder.fc
RIX Examples
FlyScan: Mean Vs. Scan Value
An example of using the MeanVsScan Box to plot an average parameter (in this case the width of a distribution) vs. the fly-scanned PV. "fly scan" means the scan value is changed independently while the DAQ free-runs. Compared to a "step scan" (see below) it has the disadvantage that there is no precise knowledge of the scan variable value at each event, since the scan variable is moving during data taking and is only recorded by the DAQ at ~1Hz. Note that this example uses a "MeanVsScan" box which can be used for both fly-scans and step-scans: for fly scans the "binned" option (and histogram limits) of the MeanVsScan box should be selected in advance.
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_flyscan.fc psana://exp=rixx43518,run=34,dir=/sdf/data/lcls/ds/prj/public01/xtc/
Waveform Sideband Subtraction
Demonstrates a 2D ROI, projection, and subtraction of average of 1D ROI of the projection.
ami-local -b 1 -f interval=1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_bkgdsub.fc psana://exp=rixx43518,run=34,dir=/sdf/data/lcls/ds/prj/public01/xtc/
Step-Scan With Gaussian Fit
A "step scan" is a mode where the DAQ is paused whenever the scan variable changes. Unlike a fly-scan, it is possible to assigned each event to a well-defined scan variable value. These scans are typically controlled by the hutch-python tool, which uses the bluesky tool from BNL. In this particular example, pedestal subtraction helps the fit significantly and is done in a kludgy way with using /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.py which generates a numpy pedestal file /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.npy. The pedestal subtraction is done with a PythonEditor box.
ami-local -b 1 -f interval=0.3 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_width_scan.fc psana://exp=rixx43518,run=45,dir=/sdf/data/lcls/ds/prj/public01/xtc/
Photon Finding and Check of Linearity
Example includes:
- photon centroid finding on vertically binned Andor camera
- linearity check of normalization (andor camera intensity vs. acqiris waveform intensity) with pearson-R calculation as a function of time
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/run22.fc hdf5:///reg/g/psdm/tutorials/ami2/neh2.2/worker.json
Waveform Integration vs. Delay Stage
Example includes:
- Plot of integration of acqiris vs delay stage
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run37.fc
Normalization of Andor
Example includes:
- 1d ROI of Andor
- Normalization of ROI by integration of acqiris
ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlr2916_run23.fc
Ratio of Peak Areas and EPICS Export of Result
Example includes:
- Ratio of peak area and export over EPICS
ami-local -e ami -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlq7615_run176.fc pvlist localhost:5075 pvget ami:ana:graph:data:peak_area
AMI2 for LCLS1 Example
Dan Damiani has been adding support for LCLS1 data in ami2. The "-b 0" option sets the heartbeat period to 0 ms and currently (3 May 2024) is required for operation.
It's still a work in progress but this example demonstrates this functionality:
# S3DF source /sdf/group/lcls/ds/ana/sw/conda1/rel/ami_current/setup_env_lcls1.sh # old psana system source /cds/group/pcds/dist/pds/ami2-devel/setup_env_lcls1.sh ami-local psana://exp=mecly4720,run=30,repeat=true -b 0
XPP Example
ami-local hdf5:///reg/g/psdm/tutorials/ami2/xpp/xpp.json
Random Data Examples
Calculator example
Example includes:
- calculator executing based on conditional logic
ami-local -l /reg/g/psdm/tutorials/ami2/random/calculator.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Complex example
Example includes:
- binning based on conditional logic
ami-local -l /reg/g/psdm/tutorials/ami2/random/complex_example.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Timestamp Plot example
Example includes:
- Plotting average of integration of waveform against time
ami-local -l /reg/g/psdm/tutorials/ami2/random/timeplot.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Export Example
Example includes:
- Averaging waveform over 100 events and exporting result from global collector to workers
- Background subtraction of averaged waveform on workers
ami-local -e test -l /reg/g/psdm/tutorials/ami2/random/export.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json
Reference Plot Example
In AMI one can export the x,y values of a waveform to a .csv file using the "camera" icon shown in the top left of the above window. That .csv file can then be loaded with a LoadReference1D box which makes those (static) values available for later analysis/display. NOTE: if you overlay the reference data with a signal in a plot be careful that the horizontal scales for both are the same.
ami-local -l /reg/g/psdm/tutorials/ami2/random/ref_waveform.fc -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/sdf/data/lcls/ds/prj/public01/xtc/
Running Multiple Independent Graphs
It is possible have a single ami instance run additional independent graphs by executing a command like this on one of the daq nodes (e.g. drp-srcf-cmp*) or control room machines like tmo-daq:
ami-client -H <ami_manager_node_name> -g <my_unique_graph_name> -p <port_number>
This will bring up the usual ami user-interface window where you can build a graph (or you can load a previously saved graph with the usual "-l" option described above). If this command hangs without bringing up an ami-graph window, it is typically a sign that ami-client cannot connect to the ami-manager host ("-H" option) and port ("-p" option).
Note that the graph name given to the "-g" parameter must be unique (the default for ami is "graph", so don't use that one). The correct hutch daq environment also has to be setup (e.g. "source /cds/group/pcds/dist/pds/tmo/scripts/setup_env.sh"). The "ami_manager_node" node name can be determined by looking at the settings in the daq .cnf file for the appropriate hutch (e.g. /cds/group/pcds/dist/pds/tmo/scripts/tmo.cnf for the tmo hutch).
The "-p" port_number can be determined by seeing what value has been passed to the ami-manager command using a command similar to the following. If no "-p" option has been given to ami-manager then a default is being used, and no port number needs to be given to ami-client above.
(ps-4.5.17) rix-daq:scripts> procmgr status rix.cnf | grep manager drp-srcf-cmp029 ami-manager_first RUNNING 53589 29205 ami-manager -p 5655 --hutch rix_first --prometheus-dir /cds/group/psdm/psdatmgr/etc/config/prom/rix -n 4 -N 1 (ps-4.5.17) rix-daq:scripts>
Low-Rate "Prompt" Mode
At high rates ami2 creates “batches” of events for processing (more efficient) but this gets sluggish when running low-rate experiments (e.g. 1Hz) because ami waits for the batch to end (defined by the heartbeat interval) before forwarding the batch through the pipeline. To address this issue Dan Damiani has added the ability to put ami in a “prompt” mode which effectively sets the batch size to 1. It is set on the ami collector process with the "-d=1" flag (depth of heartbeat event-builder buffer). This flag will cause ami to be more responsive at low rates, but function less efficiently at high rates.
Event Filtering in AMI
Often an analysis wants to treat events differently based upon something in the data. A common example is the presence of an eventcode, which may indicate the firing of a pump laser or some other condition. The selection logic can be put into a PythonEditor box, then the editor box should return python's None object to indicate the event flow is to be terminated here; returning a value (even 0) causes the flow to continue with that value and possibly yields unexpected/incorrect results. An example flowchart with a PythonEditor box doing eventcode filtering is shown below. This particular example "ping pongs" an image output between two displays depending on whether an eventcode is present or not.
This example should be run like this:
ami-local -b 1 -f interval=1 -l /sdf/group/lcls/ds/ana/tutorials/ami2/eventcode_filter.fc psana://exp=tmoc00118,run=222,dir=/sdf/data/lcls/ds/prj/public01/xtc/
The code inside the Python editor box that performs user-specified logic based on the eventcodes is:
class EventProcessor():
def __init__(self):
pass
def begin_run(self):
pass
def end_run(self):
pass
def begin_step(self, step):
pass
def end_step(self, step):
pass
def on_event(self, image, codes, *args, **kwargs):
# an eventcode that alternates for this data
if codes[14]:
return (image, None)
else:
return (None, image)
Configuring Plots
Many features of plots can be controlled from the "wrench" icon in the upper-left of AMI plots. This menu allows you to do things like:
- control auto-ranging
- plot title
- log scale
- add multiple user-defined "annotations" to plots (e.g. crosses, circles, squares, lines) by clicking on the "wrench" icon on the top left of ami plots. For example, here we have added a "plus-sign" annotation on top of an image at coordinates (20,20):
Loading Data in AMI
There are three different ways to load data in AMI, either through using the gui, providing a json file, or by passing command line arguments. The different methods are described below.
Command Line Configuration
Given a psana experiment and run number the xtc2 files can be loaded using the following syntax:
ami-local psana://exp=tstx00417,run=82
A more complex example that loads a saved flowchart, sets the "heartbeat" interval (-b) to 1 (defaults to 10) and fetches events from disk at 1Hz ("-f interval", defaults to as fast as possible) is:
ami-local -l spike.fc -b 1 -f interval=1 psana://exp=rixdaq18,run=18
The "-f init_time=NN" (where "NN" is a number of seconds) can also be important. It tells the data reading to wait and give the analysis side of ami a chance to startup. This is necessary because the data reading portion starts immediately and can drop events, which is what ami needs to do when running in realtime (to ensure that it is only looking at recent data).
hdf5 files with timestamp-aligned datasets can be analyzed with a command like this:
ami-local -f interval=0.1 -f repeat=true hdf5://files=/reg/g/psdm/tutorials/ami2/neh2.2/run22.h5
Loading Data through the GUI
The first option is to run ami-local without any flags. Once the main window appears, click the Configure button and the window in the image above will appear. From there hdf5 files or xtc2 files can be loaded by clicking on Add. Only one type of file may be loaded at a time, either hdf5 or xtc2, but multiple files can be loaded. AMI will loop through all the events in each file in the order they are loaded. The different options are explained below.
Note: If a source is configured through the gui configuration window, the source configuration will be saved with the graph and can be automatically reloaded by running ami-local -l <graph.fc>
JSON Configuration File
{
"interval": 0.01,
"init_time": 0.5,
"repeat": true,
"files": ["run22.h5"]
}
Additionally a JSON file containing a source configuration may be provided. A URI to the file where the prefix contains the source type must be passed to AMI. For example to load the above file run ami-local hdf5:///worker.json to use the psana source run ami-local psana:///worker.json.
Configuration Flags
| Flag | Description |
|---|---|
| interval | Time in seconds to sleep between events |
| init_time | Time in seconds to sleep before emitting the first event |
| repeat | Loop over files/events after reaching end |
| files | Comma separated list of files to load (either xtc2 or hdf5) |
These flags can be passed to the source using any of the three methods above, the gui configuration window, a json file, or using the -f command line flag. To specify multiple flags using the command line pass -f multiple times. For example:
ami-local -f interval=1 psana://exp=tmoc00118,run=4,dir=/ffb01/data/tmo/tmoc00118/xtc/
Viewing Low-Rate Detectors in AMI
By default, low-rate detector data can be "swamped" by high-rate events, so no updates are seen for low-rate detectors. Ric Claus provides a fancy way of selecting which events have priority to go to ami (the "monitoring event-builder (meb) decision" which can be done with python) but there is also a readout-group based mechanism available which may be more commonly used since it is simpler. Ric writes:
"Look at the trigger_0 detector in configDb. To increase the chances seeing slow detector events in AMI, you need to configure the TEB to reserve some buffers for the slow readout group triggers. This is done by entering the number of buffers you want to reserve for the readout group in the rogRsrvdBuf element for the group. One buffer may be sufficient, but you could try raising the number of reserved buffers, up to the number of buffers the MEB was configured with (64?). At the maximum, I think there would never be any buffers available for the fast readout group(s) so you would never see anything from the fast detectors in AMI."
Reporting AMI Issues
Users are encouraged to submit AMI bug reports, feature requests, and questions through github issues: https://github.com/slac-lcls/ami/issues
If appropriate, please include an experiment, run number, and example graph so AMI can be run in off-line mode to reproduce issues.
Screenshots
From the first user experiment of 2020:
References
- AMI Examples ... - more examples for Detector Geometry, Mask, RoiArch, access components of the method det.raw.calib
Overview
Content Tools