Tasks
- monitoring EB design iteration
- gasnet
- python analysis-chain mockup
Monitoring event-builder requirements
- user-defined selectable events with static criteria
- could request either full-events or partial
- maybe we want dynamic criteria so we can say give me IPM > 0.7
- note: there is commonality between trigger storage decisions and monitoring distribution decisions
Display requirements
- dynamic filtering on a value (e.g. IPM) maybe implemented as additional dimension
User Interface requirements
- programmatic interface
- click interface (x vs. y)
- do we need a box-drag interface? not clear: maybe to edit analysis, persist analysis. but maybe can push some of this power into the plot.
Chains of analyses requirements
- Use separate processes for 1 event in a reddis-like decentralized environment?
- support an arbitrary loopless graph of analysis
- support for reading/writing EPICS variables in real-time (having analysis/daq/controls be in the same python environment is necessary for this). would help with feedback
- maybe could replace hutch python to control DAQ with this? (scans)
- 1 event 1 core
- scientists only write algorithm boxes
- two levels: "box" level, and then the "scheduler"
- how do we pass meta-data through the tree? (e.g. for two ROIs in a row, the second one needs to know about the first one)
- policy for type-checking and shape-checking?
Multiprocessing
- display makes requests to clients? both algorithm/parameter changes and gather messages
- different rates for different gather requests?
- how do you guarantee consistency with changing calculations e.g. ROI?
Reduce/Gather Design Ideas
- specify what to send, when to send (upstream)
- what to accept what to do with it
- send on timer or event-count
- reduce-last-hundred-events will be fuzzy (alternative is to constrain the way daq hands out events to cores)
Plot "Boxes" Ideas
- keep them separate from the graphs
- many output boxes (plot, save to disk, etc.)
- three big ideas: management, graph, output
Reconfiguration of Graph
- reconfigure everything downstream of the box that gets changed
- boxes like a "deque" box will clear their deque's.
- boxes could have a variety of more complex reconfigures that have fewer side-effects (e.g. not clear the deques in some circumstances)
Possible Graph Backward Loops
- This is the case where a background is computed by a post-reduce box and fed to an upstream box
- We think we can avoid backward loops in this case by having one chain of post-gather boxes feed upstream boxes
Topics to Discuss
- metadata
- slow data (epics)
- data interface (top-down vs. bottom-up design)
- IOC recorders
Packages to Consider
pyqtgraph
mpi
zmq
karabo:
send mail to k. weger
vistrails.org:
matplotlib builtin
awkward
python
no obvious c/c++ code
necessary to visualize the workflow
can we group operations into a hierarchy
only qt4 python2 (no python 3)
xicam:
http://www.camera.lbl.gov/features-xi-cam
looks interesting. will talk to LBL developers.
orange:
looks interesting
recommended as possibility by Tassone/Lenson
gasnet.lbl.gov:
data transport used by software like legion
support infiniband
http://vispy.org
clemens: going slowly, recommends pyqtgraph
more for rendering with opengl
replacement for pyqtgraph
https://pypi.python.org/pypi/guidata
user-interface generation. unlikely, but maybe...
www.paraview.org:
no
c++/qt
vtk
https://wci.llnl.gov/simulation/computer-codes/visit:
c++, rendering, doesn't look too fancy
vtk
interactivity with 2D visualization not so good?
browser based
taverna:
written in java
kepler:
written in java
vtk:
3D visualization? Not so useful
http://nerds.airbnb.com/airflow/
workflow (hierarchical batch job) management in python
https://developers.google.com/protocol-buffers/
language-independent structured data. doesn't feel too useful.
epicsv4 structured channels:
more wire-protocol stuff
http://www.jmp.com/en_us/home.html
plotting
closed source
https://www.tableau.com
closed source
software to make nice dashboards
nano-surveyor (from camera people)
Overview
Content Tools