Background

When setting up AMI graphs one of the more complex issues is gathering results from the multiple nodes/cores that are receiving the events.  This gather is done at a "heartbeat" rate (typically 1-10Hz) that uses the event timestamps do determine when to determine if enough time has passed so that a gather operation should happen.

These boxes are colored blue in the AMI GUI.

Before describing how to accumulate events in AMI, it is important to understand the architecture of AMI, which consists of workers, local collectors, a global collector, a manager, and a client processes that run on different compute nodes.

The client assembles a graph which is pushed to the manager that then distributes it to the workers and collectors for execution. Nodes in the graph are "colored" worker, localCollector, and globalCollector. Each type of process is responsible for executing a subgraph of nodes of the appropriate color (ie workers execute nodes with the color worker, ...and similarly for the local collector and global collector). Events in AMI are distributed to workers by Psana. AMI provides different types of graph nodes for accumulating events across this distributed architecture. Nodes which accumulate events are called global operations and GUI graph nodes are expanded into 3 underlying graph nodes of the same type but with different colors.

Average

...

Current Global Operations in AMI

The following global operations are commonly used by users in AMI:

• Average0D, Average1D, Average2D
• Binning, Binning2D
• MeanVsScan, MeanWaveformVsScan

These global operations are typically used internally by AMI:

• Accumulator, Pick1, PickN, ReduceByKey, RollingBuffer

Any node which follows a global operation will only run on the global collector.

Average0D, Average1D, Average2D

Will average event boxes)Will average event data either over an infinite time ("infinite" checked in configuration) or a settable finite number of events (generated internally using the PickN SumN pattern).  For the finite case be careful not to set N too large as it can consume a lot of memory.  The PickN produces an array of numbers/1D-arrays/2D-arrays which are, respectively, 1D/2D/3D arrays (with "time" being the added dimension).  AMI allows you to average over any of those axes, but users will typically want to average over axis=0 ("time"), unless you want to see an average projection (axis=1 or 2) vs. time.

This pattern is similar to an MPI "reduce" operation.

Binning, Binning2D

MeanVsScan, MeanWaveformVsScan

Patterns Use Internally By AMI

PickN

Use-case summary: Use PickN when you need to gather a precise number of events from multiple workers.  It is unusual for users to directly use this pattern: it is typically only used internally by AMI.  The ami graph output will update only when N events have been collected.  Note that there is some arithmetic rounding depending on the number of events requested and the number of workers running parallelrequested and the number of workers running parallel.  This pattern (and RollingBuffer, below) are similar.

Be careful to not set N too large if the items being selected are also large (e.g. camera images) since that can use up a lot of memory in worker processes and local/global collector processes.

PickN accumulates N events into a list. The GUI node is expanded into 3 underlying PickN graph nodes each with a color of worker, localCollector, and globalCollector. The underlying graph nodes have a different N.

...

The PickN will only update when N events have been collected. There is no guarantee on the order in which events are collected (the collectors can, in some cases, throw away events that are "old", but it's not perfect).

This pattern is similar to an MPI "gather" operation (gather a list of items).

RollingBuffer

Use-case summary: Use  similar to PickN.  Use RollingBuffer like a circular buffer (or first-in-first-out buffer) that keeps a finite number of events.  The  Unlike PickN the ami graph output updates every event.

RollingBuffer works similarly to the PickN, only it accumulates to a circular buffer, which shifts left after N elements have been accumulated. Unlike the PickN which only returns when N events have collected, the RollingBuffer returns after each eventeach event.

This pattern is similar to an MPI "gather" operation (gather a list of items).

For example assume N = 10 and we have 2 workers and 2 localCollectors. The workers and local collectors will both collect 5 events each and the global collector will collect 10 events. Once 10 events have been collected all elements in the list will shift left and the latest event will appear in the buffer at index 9.

...

globalCollector: [6, 7, 8, 9, 10, 2, 3, 4, 5, 11]

Accumulator

An accumulator takes two functions as an argument, a reduction and a res_factory that is called on reset. It accumulates into self.res on each event in the following way: self.res = reduction(self.res, *args) at the end of a heartbeat workers and local collectors reset self.res by calling self.res_factory.

...

https://github.com/slac-lcls/ami/blob/master/ami/flowchart/library/Numpy.py#L506-L528

ReduceByKey

ReduceByKey takes a key and value and reduction function and reduces into a dictionary in the following way: if key in self.res: self.res[key] = reduction(self.res[key], value) else: self.res[key] = value

...

https://github.com/slac-lcls/ami/blob/master/ami/flowchart/library/Operators.py#L149-L184

Current Global Operations in AMI

The following operations in AMI use one of the above accumulations and are considered global operations:

• Average0D, Average1D, Average2D
• Binning, Binning2D
• MeanVsScan, MeanWaveformVsScan
• Accumulator, Pick1, PickN, ReduceByKey, RollingBuffer

Any node which follows a global operation will only run on the global collector.

Accumulator Example

Example includes:

• ReduceByKey
• Accumulator
• RollingBuffer
• PickN

...