Page History

Background

Before describing how to accumulate events in AMI is a distributed architecture, consisting , it is important to understand the architecture of AMI, which consists of workers, local collectors, a global collector, a manager, and a client processes that run on different compute nodes.

The client assembles a graph which is pushed to the manager that then distributes it to the workers and collectors for execution. Nodes in the graph are "colored" worker, localCollector, and globalCollector. Each type of process is responsible for executing a subgraph of nodes of the appropriate color (ie workers execute nodes with the color worker, ...). Events in AMI are distributed to workers by Psana. AMI provides different types of graph nodes for accumulating events across this distributed architecture. Nodes which accumulate events are called global operations and GUI graph nodes are expanded into 3 underlying graph nodes.

PickN

PickN accumulates N events into a list. The GUI node is expanded into 3 underlying PickN graph nodes each with a color of worker, localCollector, and globalCollector. The underlying graph nodes have a different N. Workers collect N // num_workers, localCollectors collect N // num_local_collectors, and the globalCollector collects (N // num_workers) * num_workers.

In the above example there are 4 workers and 2 localCollectors, if we wanted to collect N = 25 events then each worker would collect 25 // 4 = 6 events, each localCollector would collect 12 "events" from the workers (ie 2 lists of length 6 each),  and the globalCollector would collect (25 // 4) * 4 = 24 "events" from the localCollector (ie 2 lists of length 12 each).

The PickN will only update when N events have been collected.

RollingBuffer

RollingBuffer works similarly to the PickN, only it accumulates to a circular buffer.

Accumulator

ReduceByKey