Life of a Partition
- collection manager (CM) alive (long lived)
- restart .cnf
- python-proxies comes up
- proxies reports existence/identity (Detector/DRP, EB, monshmserverAMI, monshmserverPsana, DTI, XPM, Control) to CM
- control level queries CM for list of all processes in the platform
- control level notifies CM of processes it wants for partition
- CM tells all processes about all the nodes in the partition
- processes figure out how many "ports" (an abstract idea, could be q-pair for IB) they need given the nodes in the partition and report back their ports to CM
- CM treats ports as opaque information
- CM also manages connection information for detector-to-dti-link-to-drp-node mapping table (connection information) which is used by the DTI-proxy to compute a dti-link-mask.
- after gathering all port information, CM broadcasts all ports, as well as CM-assigned "id" to proxies
- proxies are specialized for particular levels (DRP, EB, DTI) and select ports they are interested in
- proxies make appropriate connections based on their CM-assigned ID number
- proxies report that they are connected, or failure (e.g. if DTI-link-mask is already allocated, or IB connection fails)
- after all proxies report in, manager notifies control level that transitions can be sent
- control level starts to send transitions
- graceful teardown: unconfigure/unmap transitions complete. CM sends disconnect message to proxies
- ungraceful teardown: feels like we need to restart, since things can be broken in various ways
Partition vs. Platform
- one .cnf file corresponds to one platform
- a partition is a subset of the detectors in a platform
- there cannot be two partitions per platform: instead create a second platform with an additional .cnf
Contents of .cnf file
- collection related
- EB use IB or IP (goes to Ric)
- ip of CM (determines network)
- (maybe) map of detector-to-dti-link-to-drp-node-pgp-lane
- monshmserver group IDs (e.g. psana/AMI)
- non-collection
- static detector discovery? (dynamic feels difficult)
- outfile paths
- configdb info
- shmem names
- drp algs (e.g. ROI)
Design Decisions
- one CM per .cnf (long-lived process)
- does collection control the BOS? We believe not. Doesn't change frequently
- does monshm participate in collection? yes
- how flexible are DRP exe's: ideally multiple dlopen's, so long-lived, but feels unlikely given constraints
- more seamless restarts? on-the-fly restarts feel very difficult. would complicate the code significantly. better to keep code simple. AMI can be restarted independently.
- use zmq select to timeout
- use python-proxies for C++. Communicate python-created datagrams to C++ via zmq.
Issues
- Will procmgr scale? Can we reduce restart time?
- Does map of detector-to-dti-link-to-drp-node-pgp-lanes live in cnf or database?
- How do we manage the BOS?
Overview
Content Tools