Projects
- prototype point-and-click interface
- (2) glue in psana/roi/process spawning (getting the detector list from DetNames())
- (1) GraphManager (graph synchronization)
- deleting graph pieces (reference counting)
- (3) drag-and-drop box designer
- epics for output boxes
Done
- set ROI, get json
- startup new processes for windows
Tasks
- monitoring EB design iteration
- gasnet
- python analysis-chain mockup
Monitoring event-builder requirements
- user-defined selectable events with static criteria
- could request either full-events or partial
- maybe we want dynamic criteria so we can say give me IPM > 0.7
- note: there is commonality between trigger storage decisions and monitoring distribution decisions
Display requirements
- dynamic filtering on a value (e.g. IPM) maybe implemented as additional dimension
User Interface requirements
- programmatic interface
- click interface (x vs. y)
- do we need a box-drag interface? not clear: maybe to edit analysis, persist analysis. but maybe can push some of this power into the plot.
Chains of analyses requirements
- Use separate processes for 1 event in a reddis-like decentralized environment?
- support an arbitrary loopless graph of analysis
- support for reading/writing EPICS variables in real-time (having analysis/daq/controls be in the same python environment is necessary for this). would help with feedback
- maybe could replace hutch python to control DAQ with this? (scans)
- 1 event 1 core
- scientists only write algorithm boxes
- two levels: "box" level, and then the "scheduler"
- how do we pass meta-data through the tree? (e.g. for two ROIs in a row, the second one needs to know about the first one)
- policy for type-checking and shape-checking?
- each analysis box could have optional "output box" for distribution of results
Multiprocessing
- display makes requests to clients? both algorithm/parameter changes and gather messages
- different rates for different gather requests?
- how do you guarantee consistency with changing calculations e.g. ROI?
Reduce/Gather Design Ideas
- specify what to send, when to send (upstream)
- what to accept what to do with it
- send on timer or event-count
- reduce-last-hundred-events will be fuzzy (alternative is to constrain the way daq hands out events to cores)
Plot "Boxes" Ideas
- keep them separate from the graphs
- many output boxes (plot, save to disk, etc.)
- three big ideas: management, graph, output
- could be epics
Reconfiguration of Graph
- reconfigure everything downstream of the box that gets changed
- boxes like a "deque" box will clear their deque's.
- boxes could have a variety of more complex reconfigures that have fewer side-effects (e.g. not clear the deques in some circumstances)
Possible Graph Backward Loops
- This is the case where a background is computed by a post-reduce box and fed to an upstream box
- We think we can avoid backward loops in this case by having one chain of post-gather boxes feed upstream boxes
Epics
- Feel like it should be split out of the DAQ timestamped data
- recorded separately to hdf5 and handled by offline event builder
- for shmem another process would put epics in some separate buffer (like "epicsStore") for use by AMI
- could be used to send ami data elsewhere (from output boxes)
Data Interface
- boxes accept "datagrams" as input/output
- datagrams contain: evt ids (plural, must be >0), ami-graph-config number, metadata dictionary which has serializable python objects, data dictionary of numpy arrays
IOC Recorders
- hope these don't need to be processed by AMI, unless DAQ manages to bring them into the hutch-DAQ via "PGP broadcast"
- if they produce files like current system, AMI would handle them with psana offline event builder
- as alternative to PGP, online event-builder could grab well-timestamped epics data for us in AMI
DAQ/DRP Reconfig
- we will assume the DAQ/DRP reconfigs look identical to us
- DRP reconfigs can be generated from AMI
- we think DAQ/DRP reconfigs don't interfere with AMI-analysis-graph reconfigs
Display Images at Low Rate
- no "requests" to graph from display clients
- use "reduce" boxes
Monitoring of AMI Performance/Health
- stats sent when data flowing
- isAlive-health monitored with procserv like process
- ideally could trigger a python-strace when stuck
- stats do not flow through the data graph: separate path (a separate stats piece of the graph?)
DAQ Transitions
- Try to eliminate transitions in the DAQ
- instead use "base-configuration" for a run, and time-ranges-of-validity for configuration changes (scans)
- online and offline event-builders will event-build the configuration changes accordingly
Output Boxes:
- could be EPICS? (and others)
- slow data (e.g. slowly changing background) can be posted back to earlier parts in the graph in a slow "database" fashion
Posting Data to Other Calculations
- All data can be posted
- Slow data (e.g. slowly changing background) can be posted back to earlier parts of the graph
Names
- Each box output gets a "name" that users can use to access it for further computation
- bigger boxes encompassing more steps mean fewer names
pyDM
- could be useful for assembling AMI windows
- AMI is made up of relatively few "building block" windows
- could also be useful for standard-config
To Discuss
- changing data as run goes on (e.g. DRP pre scaled uncompressed events)
Packages to Consider
dasc
pyqtgraph
mpi
zmq
karabo:
send mail to k. weger
vistrails.org:
matplotlib builtin
awkward
python
no obvious c/c++ code
necessary to visualize the workflow
can we group operations into a hierarchy
only qt4 python2 (no python 3)
xicam:
http://www.camera.lbl.gov/features-xi-cam
looks interesting. will talk to LBL developers.
orange:
looks interesting
recommended as possibility by Tassone/Lenson
gasnet.lbl.gov:
data transport used by software like legion
support infiniband
http://vispy.org
clemens: going slowly, recommends pyqtgraph
more for rendering with opengl
replacement for pyqtgraph
https://pypi.python.org/pypi/guidata
user-interface generation. unlikely, but maybe...
www.paraview.org:
no
c++/qt
vtk
https://wci.llnl.gov/simulation/computer-codes/visit:
c++, rendering, doesn't look too fancy
vtk
interactivity with 2D visualization not so good?
browser based
taverna:
written in java
kepler:
written in java
vtk:
3D visualization? Not so useful
http://nerds.airbnb.com/airflow/
workflow (hierarchical batch job) management in python
https://developers.google.com/protocol-buffers/
language-independent structured data. doesn't feel too useful.
epicsv4 structured channels:
more wire-protocol stuff
http://www.jmp.com/en_us/home.html
plotting
closed source
https://www.tableau.com
closed source
software to make nice dashboards
nano-surveyor (from camera people)