Python Psana2 Interface

A shared memory client is instantiated using the psana DataSource class.

The arguments to the class constructor are as follows:

DataSource('shmem','shmem_tag')

Where 'shmem_tag' is a partition tag string shared with an associated server.

The following code example:

  • creates a shared memory datasource with the tag 'shmem_test_0'
  • connects and waits for events from the server
  • iterates and counts events
  • prints the transition ID for each datagram
  • frees datagrams back to the server
from psana import DataSource
ds = DataSource('shmem','shmem_test_0')
events = 0
run = next(ds.runs())
for evt in run.events():
    print("event",events,"transitionId",evt._dgrams[0].seq.service())
    del evt._dgrams[0]
    events += 1

Runtime Behavior With Psana2

When a shmem datasource is created, it will immediately block and poll for a connection to a local server on default port 32768.

Once the connection is established, the client blocks again until it receives the first datagram with the Configure transition.

After the Configure datagram is received, the datasource run iterator calls to events() will block until the next datagram is received from the server.

A single datagram is contained in each run event.

When a datagram is deleted, the shared buffer is released back to the server.  In python, this occurs when the object reference count goes to zero.

If the connection to a server is lost, the client will not attempt to reconnect.

If the connection was not terminated properly, the client will throw an AssertionError for a NULL buffer view.

Server Transition Cache

The server monitors and caches state transitions as determined by the transition ID contained in each datagram.

When a connection to the server is made, any cached transition datagrams, such as Configure or Enable, will be immediately delivered to the client.

These cached transition datagrams are sent to clients prior to any L1 Accept datagrams.

C++ Interface

The C++ interface to the shared memory client consists of three methods defined in the ShmemClient class: connect, get, and free.

This class is made accessible to python and psana2 using the cython wrapper class PyShmemClient.

The connect method establishes communication paths to the server.

The get method is blocking and returns the next shared memory buffer available from the server.

This get method operates similar to a linear file read for easier integration with single-file psana.

When a connection is lost, or a fatal error occurs, a NULL buffer will be returned from the get method.

The free method releases ownership of a shared memory buffer back to the server.

Client Server Executables

Both the client and the server utilize executables for test and debug purposes.

 

Usage: shmemClient [-p <partitionTag>] [-i <index>] [-r <rate>] [-h]
Usage: shmemServer -p <partitionTag> -n <numberOfBuffers> -s <sizeOfBuffers> [input_options] [other_options]
input_options:
 -f <filename> : full path of xtc file
 -l <filename_list> : name of file contains list of xtc file paths
 -x <run_file_prefix> : path prefix to match (e.g. e302-r0008)
 -d <xtc_dir> : directory containing all xtcs to use
other_options:
 [-r <ratePerSec>] [-c <# clients>]
 [-L] : (continuous loop)
[-v] [-V]

 

In order to run a server and client (DataSource or executable) test, a proper xtc2 file must be used as input to the server.

Client and server startup ordering is not required, as both will independently poll for connection requests.

PyTest Module

The shmem client pytest consists of the following files in psana/psana/test:

  • shmem_client.py - connects to server, waits for datagrams, counts L1 accept transitions
  • test_shmem.py - spawns a server and 4 clients, then transfers 64 L1 event datagrams to each client

Source Code

The shared memory C++ source can be found in psalg/psalg/shmem/:

  • ShmemClient.cc - client base class
  • ShmemClient.hh - client interface
  • shmemClientApp.cc - example C++ client executable
  • shmemServer.cc - server executable
  • XtcMonitorServer.cc - server base class
  • XtcMonitorServer.hh - server interface

The shared memory python source is in psana/:

  • psana/shmem/shmem.pyx - Cython bindings for client C++ class
  • psana/psexp/shmem_ds.py - Datasource interface to client
  • psana/dgrammanager.py - Implements shmem client control and transfer of datagrams 
  • src/dgram.cc - Implements creation of datagrams using shmem client buffers 

Server Buffer Logic

Some notes on how the shmem server works, after discussions with Chris Ford, Chris O'Grady, Matt Weaver on July 31, 2019.

transitions go through tcp to the client, with the index of the shmem buffer
l1accepts:

  • go to the server shuffle queue (_pfd[2])
  • server runs some pnccdshuffle code
  • XtcMonitorServer::events() sends a buffer number to the client by calling XtcMonitorServer::_send() which posts to the "shuffle" message queue
  • l1accepts can be dropped in XtcMonitorServer::_send() if no free-buffer number is received from a client from the message queue
  • shuffle message queue is poll'd by the XtcMonitorServer::routine() thread, which copies the dgram into the shmem buffer
  • tcp is used for discover() and to detect when a client goes away
  • also see the comments on the top of XtcMonitorServer.hh
  • there are two server modes: distributing events across clients or broadcasting each event to all clients.  this is controlled by the XtcMonitorServer::distribute() call ("-d" option to monReqServer, but not an option for the file-based shmemServer)

_pfd[0] is a message from the discover thread that new client exists
_pfd[1] buffers returned from client
_pfd[2] events from the eb (ric) (gives a chance for pnccdshuffle, for example)
_pfd past 2 is for client tcp connections to receive transitions and reply "complete"

feel like Ric's monEb calls XtcMonitorServer::events(dgram) method to inject

In this picture, each "square" represents a thread or process:

 

  • No labels