Page History

...

PickN accumulates N events into a list. The GUI node is expanded into 3 underlying PickN graph nodes each with a color of worker, localCollector, and globalCollector. The underlying graph nodes have a different N.

Workers collect: N // num_workers

, localCollectors collect: N // num_local_collectors

, and the globalCollector collects: (N // num_workers) * num_workers.

In the above example there are 4 workers and 2 localCollectors, if we wanted to collect N = 25 ~25 events then each worker would collect 25 // 4 = 6 events, each localCollector would collect 12 "events" from the workers (ie 2 lists of length 6 each),  and the globalCollector would collect (25 // 4) * 4 = 24 "events" from the localCollector localCollectors (ie 2 lists of length 12 each).

The PickN will only update when N events have been collected. There is no guarantee on the order in which events are collected. 

RollingBuffer

RollingBuffer works similarly to the PickN, only it accumulates to a circular buffer, which shifts left after N elements have been accumulated. Intermediate results are returned.

For example assume N = 10 and we have 2 workers and 2 localCollectors. The workers and local collectors will both collect 5 events each and the global collector will collect 10 events. Once 10 events have been collected all elements in the list will shift left and the latest event will appear in the buffer at index 9.

worker1 buffer: [1, 2, 3, 4, 5]

worker2 buffer: [6, 7, 8, 9, 10]

localCollector1 buffer: [1, 2, 3, 4, 5]

localCollector2 buffer: [6, 7, 8, 9, 10]

globalCollector: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

worker1 receives NEW EVENT 11

worker1 buffer: [2, 3, 4, 5, 11]

worker2 buffer: [5, 6, 7, 8, 9, 10]

localCollector1 buffer: [2, 3, 4, 5, 11]

localCollector2 buffer: [5, 6, 7, 8, 9, 10]

globalCollector: [6, 7, 8, 9, 10, 2, 3, 4, 5, 11]

Accumulator

ReduceByKey