View Case

There are 3 "view" dgram memory managers where the underlying memory is managed by an external entity (not Dgram.cc):

  • smd0 chunks (Mona has a DgramLite class)
  • eb chunks
  • bd chunks

Here there is no free of the dgram memory and this code is called: https://github.com/slac-lcls/lcls2/blob/73283e720cf9d2930520daee5b2077e5ba6188fc/psana/src/dgram.cc#L557.  This is the common case used for the standard exp=tmox43219,run=321, even if we filter events in EB level.  This will also be used for the python-drp dgram memory management (to avoid freeing memory in the pebble).

In many cases where we use a view, the view object is a "batch" of dgrams.  So if the user holds on to a reference to an image in one dgram, that increments the reference count for the view ().  The whole batch can have a long lifetime in this scenario.

Detailed View Case for BD Nodes

batch
dg1 (view1, e.g. offset 0 bytes into pread buffer)
dg2 (view2, e.g. offset 50 bytes)
dg3 (view3, e.g. offset 80 bytes)

memory for the batch is allocated by the top level owner like this (psana/psana/psexp/event_manager.py):

            new_read = os.pread(fd, size, offset) # the "pread" buffer: a whole batch of dgrams

get the views for individual dgrams with this line:

                view = self.bd_bufs[i_smd]  # one dgram at offset inside pread buffer (new_read object), increments refcnt of new_read

our picture: the dgram "views" inside the pread buffer increment the refcnt of the pread buffer (similar to how the arrays in a dgram increment the refcnt of the dgram)

make a dgram with this line:

                dgrams[i_smd] = dgram.Dgram(config=self.dm.configs[i_smd], view=view, offset=offset)  # increments the refcnt for the view

aside comment: inside the dgram we have arrays that increment the refcnt for the dgram (how we avoid copying array data) with the line:

PyArray_SetBaseObject((PyArrayObject*)newobj, pyDgram→dgrambytes)

simple overall picture of hierarchical refcnt management from pread buffer to low level arrays (e.g. on a bigdata node, most complex case?)

pread->multipleViews->dgram(one-per-view)->multipleArrays

feels like similar refcnt pattern between pread->multipleViews and dgram→multipleArrays (lower-level objects keep upper-level objects alive)

because view approach in dgram.cc is using PyObject_GetBuffer and PyBuffer_Release in a "standard" way then the memory management should just work.

Non-View Case

Used for individual datagrams in dgram.cc.

For everything that is not a "view" then Dgram.cc does the dgrambytes memory management with python reference counting as described in this this line: https://github.com/slac-lcls/lcls2/blob/73283e720cf9d2930520daee5b2077e5ba6188fc/psana/src/dgram.cc#L692.  Here the dgram frees memory in this code: https://github.com/slac-lcls/lcls2/blob/73283e720cf9d2930520daee5b2077e5ba6188fc/psana/src/dgram.cc#L549

The places where we use the non-view case are:

Editing Datagrams With DgramEdit

When editing datagrams with dgramedit two buffers are used:  additions to the datagram are done by appending them to the "input datagram buffer".  There must be additional space available for these additions, and the boundaries should be set using the xtc "bufEnd" parameter to avoid obscure segfaults.  Then when the save() method is called the relevant parts of the datagram are copied to the output datagram buffer (the argument of save()) which also needs enough space allocated and an appropriate setting of "bufEnd".  This save() step is when the discarded parts of the datagram (e.g. large raw data) is removed from the datagram.

  • No labels