MPI Package Data Structure

In MPI version of PSANA2, we need to send SmallData from SMD0 to other types of node (see flowchart). These data are packaged in a way that they can be send in one message. The purpose of this article is to describe the structure of the data as appeared in the message between two layers (SMD0→EventBuilder and EventBuilder → BigData).

Structure of Dgram and Event

A Dgram consists of a Header and a Payload. Currently a header is of fixed size (24 bytes) and we can retrieve the size of the payload from the header.

An Event consists of one or more dgrams; each represents part of an event from one stream file (-sNNN). All dgrams from one event have the same timestamp. For L1Accept, missing dgrams is allowed. However, for other transitions, all must exist.

Smd0 Message Structure

Smd0 sends chunks of corresponding dgrams from different stream files in one message. To be able to this, Smd0 puts each chunk next to each other and append a footer at the end. The footer describes how many chunks are there in the message and what are their size. Below is an example of this type of message for two stream files prepared to be sent to an EventBuilder node.

To inspect the content of this message, you can use PacketFooter library to separate each chunk.

from psana.psexp import PacketFooter
from psana.dgram import Dgram
pf = PacketFooter(view=smd0_message) 		# smd0_message is any bytearray with the above structure
chunks = pf.split_packets()
offsets = [0] * pf.n_packets			    # this is used to move the pointer when creating a dgram from a chunk
for i, chunk in enumerate(chunks):
    while offsets[i] < pf.get_size(i)
        # Creates a dgram from this chunk at the given offset.
        # Note that you'll need the correct config to do this!.
        d = Dgram(view=chunk, config=configs[i], offset=offsets[i])
        print(d.timestamp(), d.service())
        offsets[i] += d._size