...

ssh psana
source /cdssdf/group/swlcls/ds/ana/sw/conda2/manage/bin/psconda.sh

...

NOTE: when learning, or developing new graphs it can be useful to slow ami down to 1Hz, using the "-b" and "-f" options shown in the first example.   "-b" is a "heartBeat" interval in milliseconds (using the timestamps of the events being read!) and controls how often the plots update.  "-f interval=1" controls the interval (in seconds) between events read from disk.

Viewing Images

• Login to a psana node and setup the environment as shown above, then type:

ami-local -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/cdssdf/data/psdmlcls/ds/prj/public01/xtc/

• When the AMI window comes up in the "search sources" box on the top left type "opal".  That will select only detectors that have the string "opal" in them
• drag the text "tmo_atmopal:raw:image" into the rightmost part of the ami window, a red box will appear
• click on the red box. a Pyqtgraph window should appear with the image, which will update at 1Hz.  The plot has many user-settable attributes.  You can try clicking the "wrench" icon in the upper left and removing the Auto-Range and Auto-Levels settings (followed by "Apply"): this will give you more control of the color scale using the horizontal yellow control bars on the right.

...

A 2D-ROI, summed over pixels.  Mean value of pixel-sum is plotted vs. step_value.  The "-b" and "-f" flags slow down the data so it doesn't go by too quickly (there are unfortunately some race conditions in ami between the start of the analysis graph and the reading of the data).

ami-local -b 100 -l /reg/g/f interval=0.002 -l /reg/g/psdm/tutorials/ami2/neh2.2/epixroi_meanvscan.fc psana://exp=rixx45619,run=122,dir=/cdssdf/data/lcls/psdmds/prj/public01/xtc//

Image reconstruction using geometry

• Geometry and Detector Image

Mask and RoiArch examples

• AMI Examples for Detector Geometry Mask and RoiArch

TMO Examples

Example includes:

...

ami-local -f repeat=true -f interval=0.1 -f init=1 -l /reg/g/psdm/tutorials/ami2/tmo/blobfinder.fc psana://exp=tmoc00518,run=1,dir=/sdf/data/lcls/ds/prj/public01/xtc//

HitFinder example

ami-local -l /reg/g/psdm/tutorials/ami2/tmo/hitfinder.fc

...

ami-local -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_flyscan.fc psana://exp=rixx43518,run=34,dir=/cdssdf/data/psdmlcls/ds/prj/public01/xtc/

Waveform Sideband Subtraction

...

ami-local -b 1 -f interval=1 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_bkgdsub.fc psana://exp=rixx43518,run=34,dir=/cdssdf/data/lcls/psdmds/prj/public01/xtc/

Step-Scan With Gaussian Fit

...

A "step scan" is a mode where the DAQ is paused whenever the scan variable changes.  Unlike a fly-scan, it is possible to assigned each event to a well-defined scan variable value.  These scans are typically controlled by the hutch-python tool, which uses the bluesky tool from BNL.  In this particular example, pedestal subtraction helps the fit significantly and is done in a kludgy way with using /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.py which generates a numpy pedestal file /reg/g/psdm/tutorials/ami2/neh2.2/manta_ped.npy.  The pedestal subtraction is done with a PythonEditor box.

ami-local -b 1 -f interval=0.13 -l /reg/g/psdm/tutorials/ami2/neh2.2/rix_width_scan.fc psana://exp=rixx43518,run=45,dir=/cdssdf/data/lcls/psdmds/prj/public01/xtc/

Photon Finding and Check of Linearity

...

ami-local -e ami -l /reg/g/psdm/tutorials/ami2/neh2.2/sxrlq7615_run176.fc
pvlist localhost:5075
pvget ami:ana:graph:data:peak_area

XPP Example

 ami-local hdf5:///reg/g/psdm/tutorials/ami2/xpp/xpp.json

Random Data Examples

Calculator example

Image Removed

Example includes:

...

AMI2 for LCLS1 Example

Dan Damiani has been adding support for LCLS1 data in ami2.  The "-b 0" option sets the heartbeat period to 0 ms and currently (3 May 2024) is required for operation.

It's still a work in progress but this example demonstrates this functionality:

# S3DF
source /sdf/group/lcls/ds/ana/sw/conda1/rel/ami_current/setup_env_lcls1.sh
# old psana system
source /cds/group/pcds/dist/pds/ami2-devel/setup_env_lcls1.sh

ami-local psana://exp=mecly4720,run=30,repeat=true -b 0

Image Added

XPP Example

 ami-local hdf5

 ami-local -l /reg/g/psdm/tutorials/ami2/random/calculator.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Complex example

Image Removed

 Example includes:

• binning based on conditional logic

ami-local -l /reg/g/psdm/tutorials/ami2/random/complex_example.fc random:///reg/g/psdm/tutorials/ami2/randomxpp/workerxpp.json

...

Random Data Examples

Calculator example

Image RemovedImage Added

Example includes:

• calculator executing based on conditional logicPlotting average of integration of waveform against time
  
  

ami ami-local -l /reg/g/psdm/tutorials/ami2/random/timeplotcalculator.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

...

Complex example

Image RemovedImage Added

 Example includes:

• Averaging waveform over 100 events and exporting result from global collector to workers
• Background subtraction of averaged waveform on workersbinning based on conditional logic

ami-local -l /reg/g/psdm/tutorials/ami2/random/complex_example.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Timestamp Plot example

Image Added

Example includes:

• Plotting average of integration of waveform against time

ami-local e test -l /reg/g/psdm/tutorials/ami2/random/exporttimeplot.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Event Filtering in AMI

Often an analysis wants to treat events differently based upon something in the data.  A common example is the presence of an eventcode, which may indicate the firing of a pump laser or some other condition.  If the selection logic is put into a PythonEditor box, then the editor box should return python's None object to indicate the event flow is to be terminated here; returning a value (even 0) causes the flow to continue with that value and possibly weighting your statistics in unexpected ways.  An example flowchart and PythonEditor box is shown below.  Other boxes can also handle filtering with their "conditional" attributes.

Image RemovedImage Removed

Loading Data in AMI

There are three different ways to load data in AMI, either through using the gui, providing a json file, or by passing command line arguments. The different methods are described below.

Command Line Configuration

Given a psana experiment and run number the xtc2 files can be loaded using the following syntax:

ami-local psana://exp=tstx00417,run=82

A more complex example that loads a saved flowchart, sets the "heartbeat" interval (-b) to 1 (defaults to 10) and fetches events from disk at 1Hz ("-f interval", defaults to as fast as possible) is:

ami-local -l spike.fc -b 1 -f interval=1 psana://exp=rixdaq18,run=18

The "-f init_time=NN" (where "NN" is a number of seconds) can also be important.  It tells the data reading to wait and give the analysis side of ami a chance to startup.  This is necessary because the data reading portion starts immediately and can drop events, which is what ami needs to do when running in realtime (to ensure that it is only looking at recent data).

Loading Data through the GUI

Image Removed

The first option is to run ami-local without any flags. Once the main window appears, click the Configure button and the window in the image above will appear. From there hdf5 files or xtc2 files can be loaded by clicking on Add. Only one type of file may be loaded at a time, either hdf5 or xtc2, but multiple files can be loaded. AMI will loop through all the events in each file in the order they are loaded. The different options are explained below.

Note: If a source is configured through the gui configuration window, the source configuration will be saved with the graph and can be automatically reloaded by running ami-local -l <graph.fc>

JSON Configuration File

 {
    "interval": 0.01,
    "init_time": 0.5,
    "repeat": true,
	"files": ["run22.h5"]
}

...

Configuration Flags

...

Export Example

Image Added

Example includes:

• Averaging waveform over 100 events and exporting result from global collector to workers
• Background subtraction of averaged waveform on workers

ami-local -e test -l /reg/g/psdm/tutorials/ami2/random/export.fc random:///reg/g/psdm/tutorials/ami2/random/worker.json

Reference Plot Example

Image Added

In AMI one can export the x,y values of a waveform to a .csv file using the "camera" icon shown in the top left of the above window.  That .csv file can then be loaded with a LoadReference1D box which makes those (static) values available for later analysis/display.  NOTE: if you overlay the reference data with a signal in a plot be careful that the horizontal scales for both are the same.

ami-local -l /reg/g/psdm/tutorials/ami2/random/ref_waveform.fc -b 1 -f interval=1 psana://exp=tmoc00118,run=222,dir=/sdf/data/lcls/ds/prj/public01/xtc/

Running Multiple Independent Graphs

It is possible have a single ami instance run additional independent graphs by executing a command like this on one of the daq nodes (e.g. drp-srcf-cmp*) or control room machines like tmo-daq:

ami-client -H <ami_manager_node_name> -g <my_unique_graph_name> -p <port_number>

This will bring up the usual ami user-interface window where you can build a graph (or you can load a previously saved graph with the usual "-l" option described above).  If this command hangs without bringing up an ami-graph window, it is typically a sign that ami-client cannot connect to the ami-manager host ("-H" option) and port ("-p" option).

Note that the graph name given to the "-g" parameter must be unique (the default for ami is "graph", so don't use that one).  The correct hutch daq environment also has to be setup (e.g. "source /cds/group/pcds/dist/pds/tmo/scripts/setup_env.sh").  The "ami_manager_node" node name can be determined by looking at the settings in the daq .cnf file for the appropriate hutch (e.g. /cds/group/pcds/dist/pds/tmo/scripts/tmo.cnf for the tmo hutch). 

The "-p" port_number can be determined by seeing what value has been passed to the ami-manager command using a command similar to the following.  If no "-p" option has been given to ami-manager then a default is being used, and no port number needs to be given to ami-client above.

(ps-4.5.17) rix-daq:scripts> procmgr status rix.cnf | grep manager
drp-srcf-cmp029 ami-manager_first RUNNING    53589   29205  ami-manager -p 5655 --hutch rix_first --prometheus-dir /cds/group/psdm/psdatmgr/etc/config/prom/rix -n 4 -N 1
(ps-4.5.17) rix-daq:scripts> 

Low-Rate "Prompt" Mode

At high rates ami2 creates “batches” of events for processing (more efficient) but this gets sluggish when running low-rate experiments (e.g. 1Hz) because ami waits for the batch to end (defined by the heartbeat interval) before forwarding the batch through the pipeline.  To address this issue Dan Damiani has added the ability to put ami in a “prompt” mode which effectively sets the batch size to 1.  It is set on the ami collector process with the "-d=1" flag (depth of heartbeat event-builder buffer).  This flag will cause ami to be more responsive at low rates, but function less efficiently at high rates.

Event Filtering in AMI

Often an analysis wants to treat events differently based upon something in the data.  A common example is the presence of an eventcode, which may indicate the firing of a pump laser or some other condition.  The selection logic can be put into a PythonEditor box, then the editor box should return python's None object to indicate the event flow is to be terminated here; returning a value (even 0) causes the flow to continue with that value and possibly yields unexpected/incorrect results.  An example flowchart with a PythonEditor box doing eventcode filtering is shown below.  This particular example "ping pongs" an image output between two displays depending on whether an eventcode is present or not.

This example should be run like this:

ami-local -b 1 -f interval=1 -l /sdf/group/lcls/ds/ana/tutorials/ami2/eventcode_filter.fc psana://exp=tmoc00118,run=222,dir=/sdf/data/lcls/ds/prj/public01/xtc/

Image Added

The code inside the Python editor box that performs user-specified logic based on the eventcodes is:

class EventProcessor():

    def __init__(self):
        pass

    def begin_run(self):
        pass

    def end_run(self):
        pass

    def begin_step(self, step):
        pass

    def end_step(self, step):
        pass

    def on_event(self, image, codes, *args, **kwargs):
        # an eventcode that alternates for this data
        if codes[14]:
            return (image, None)
        else:
            return (None, image)

Configuring Plots

Many features of plots can be controlled from the "wrench" icon in the upper-left of AMI plots.  This menu allows you to do things like:

• control auto-ranging
• plot title
• log scale
• add multiple user-defined "annotations" to plots (e.g. crosses, circles, squares, lines) by clicking on the "wrench" icon on the top left of ami plots.  For example, here we have added a "plus-sign" annotation on top of an image at coordinates (20,20):

Image Added

Loading Data in AMI

There are three different ways to load data in AMI, either through using the gui, providing a json file, or by passing command line arguments. The different methods are described below.

Command Line Configuration

Given a psana experiment and run number the xtc2 files can be loaded using the following syntax:

ami-local psana://exp=tstx00417,run=82

A more complex example that loads a saved flowchart, sets the "heartbeat" interval (-b) to 1 (defaults to 10) and fetches events from disk at 1Hz ("-f interval", defaults to as fast as possible) is:

ami-local -l spike.fc -b 1 -f interval=1 psana://exp=rixdaq18,run=18

The "-f init_time=NN" (where "NN" is a number of seconds) can also be important.  It tells the data reading to wait and give the analysis side of ami a chance to startup.  This is necessary because the data reading portion starts immediately and can drop events, which is what ami needs to do when running in realtime (to ensure that it is only looking at recent data).

hdf5 files with timestamp-aligned datasets can be analyzed with a command like this:

ami-local -f interval=0.1 -f repeat=true hdf5://files=/reg/g/psdm/tutorials/ami2/neh2.2/run22.h5

Loading Data through the GUI

Image Added

The first option is to run ami-local without any flags. Once the main window appears, click the Configure button and the window in the image above will appear. From there hdf5 files or xtc2 files can be loaded by clicking on Add. Only one type of file may be loaded at a time, either hdf5 or xtc2, but multiple files can be loaded. AMI will loop through all the events in each file in the order they are loaded. The different options are explained below.

Note: If a source is configured through the gui configuration window, the source configuration will be saved with the graph and can be automatically reloaded by running ami-local -l <graph.fc>

JSON Configuration File

 {
    "interval": 0.01,
    "init_time": 0.5,
    "repeat": true,
	"files": ["run22.h5"]
}

Additionally a JSON file containing a source configuration may be provided. A URI to the file where the prefix contains the source type must be passed to AMI. For example to load the above file run ami-local hdf5:///worker.json to use the psana source run ami-local psana:///worker.json. 

Configuration Flags

These flags can be passed to the source using any of the three methods above, the gui configuration window, a json file, or using the -f command line flag. To specify multiple flags using the command line pass -f multiple times.  For example:

ami-local -f interval=1 psana://exp=tmoc00118,run=4,dir=/ffb01/data/tmo/tmoc00118/xtc/

Viewing Low-Rate Detectors in AMI

By default, low-rate detector data can be "swamped" by high-rate events, so no updates are seen for low-rate detectors.  Ric Claus provides a fancy way of selecting which events have priority to go to ami (the "monitoring event-builder (meb) decision" which can be done with python) but there is also a readout-group based mechanism available which may be more commonly used since it is simpler.  Ric writes:

"Look at the trigger_0 detector in configDb.  To increase the chances seeing slow detector events in AMI, you need to configure the TEB to reserve some buffers for the slow readout group triggers.  This is done by entering the number of buffers you want to reserve for the readout group in the rogRsrvdBuf element for the group.  One buffer may be sufficient, but you could try raising the number of reserved buffers, up to the number of buffers the MEB was configured with (64?).  At the maximum, I think there would never be any buffers available for the fast readout group(s) so you would never see anything from the fast detectors in AMI."

These flags can be passed to the source using any of the three methods above, the gui configuration window, a json file, or using the -f command line flag. To specify multiple flags using the command line pass -f multiple times.  For example:

...

Reporting AMI Issues

Users are encouraged to submit AMI bug reports, feature requests, and questions through github issues: https://github.com/slac-lcls/ami/issues

...

From the first user experiment of 2020:

References

• AMI Examples ... - more examples for Detector Geometry, Mask, RoiArch, access components of the method det.raw.calib