MPI Package Data Structure

In MPI version of PSANA2, we need to send SmallData from SMD0 to other types of node (see flowchart). These data are packaged in a way that they can be sent in one message. The purpose of this article is to describe the structure of the data as appeared in the message between two layers (SMD0→EventBuilder and EventBuilder → BigData).

Structure of Dgram and Event

A Dgram consists of a Header and a Payload. Currently a header is of fixed size (24 bytes) and we can retrieve the size of the payload from the header.

\| HEADER (24 bytes) \| ----------- Payload (N bytes)--------------\|

An Event consists of one or more dgrams; each represents part of an event from one stream file (-sNNN). All dgrams from one event have the same timestamp. For L1Accept, missing dgrams is allowed. However, for other transitions, all must exist.

Smd0 Message Structure

Smd0 sends chunks of corresponding dgrams from different stream files in one message. To be able to this, Smd0 puts each chunk next to each other and append a footer at the end. The footer describes how many chunks are there in the message and what are their size. Below is an example of this type of message for two stream files prepared to be sent to an EventBuilder node.

[ Portion from s000 ]

BS	EN	L1	L2	SU	L3	L4

[ Portion from s001 (L2 and L4 are missing)]

BS	EN	L1	SU	L3

[ Footer ]

size of s000 chunk	size of s001 chunk	2 (no. of chunks)

See repack_parallel for the portion of code that creates this structure.

To inspect the content of this message, you can use PacketFooter library to separate each chunk.

from psana.psexp import PacketFooter
from psana.dgram import Dgram
pf = PacketFooter(view=smd0_message) 		# smd0_message is any bytearray with the above structure
chunks = pf.split_packets()
offsets = [0] * pf.n_packets			    # this is used to move the pointer when creating a dgram from a chunk
for i, chunk in enumerate(chunks):
    while offsets[i] < pf.get_size(i)
        # Creates a dgram from this chunk at the given offset.
        # Note that you'll need the correct config to do this!.
        d = Dgram(view=chunk, config=configs[i], offset=offsets[i])
        print(d.timestamp(), d.service())
        offsets[i] += d._size

EventBuilder Message Structure

EventBuilders send "events" to BigData nodes. Note that content of EventBuilder's message is different from Smd0's. This is because EventBuilders build a list of dgrams into an Event Bytes prior to sending it. Below is an example of an "Event Bytes" and the complete EventBuilder message for two stream files when batch_size is 4.

Event Bytes

An example of BeginStep Event Bytes

BS-s000	BS-s001	size of d0	size of d1	2 (no. of dgrams)
		Footer

An example of L1Accept Event Bytes with one missing dgram

L2-s000	size of d0	0	2 (no. of dgrams)
	Footer

EventBuilder Message

An EventBuilder message is made of Event Bytes with a footer which indicates the sizes and the no. of Event Bytes in the message.

BS Event Bytes	EN Event Bytes	L1 Event Bytes	L2 Event Bytes	size of A	size of B	size of C	size of D	4 (no. of Event Bytes)
(A)	(B)	(C)	(D)	Footer

See repack_for_bd for the portion of code that creates this structure.

To inspect the content of this message,

from psana.psexp import PacketFooter
from psana.event import Event
pf = PacketFooter(view=eb_message) 				# eb_message is any bytearray with the above structure
event_bytes_list = pf.split_packets()
for i_evt, event_bytes in enumerate(event_bytes_list):
    # You need configs here! (try configs = run.configs)
    evt = Event._from_bytes(configs, event_bytes)
    print(evt.timestamp, evt.service)

Page tree