Projects

  • prototype point-and-click interface
  • (2) glue in psana/roi/process spawning (getting the detector list from DetNames())
  • (1) GraphManager (graph synchronization)
  • deleting graph pieces (reference counting)
  • (3) drag-and-drop box designer
  • epics for output boxes

Done

  • set ROI, get json
  • startup new processes for windows

Tasks

  • monitoring EB design iteration
  • gasnet
  • python analysis-chain mockup

Monitoring event-builder requirements

  • user-defined selectable events with static criteria
  • could request either full-events or partial
  • maybe we want dynamic criteria so we can say give me IPM > 0.7
  • note: there is commonality between trigger storage decisions and monitoring distribution decisions

Display requirements

  • dynamic filtering on a value (e.g. IPM) maybe implemented as additional dimension

User Interface requirements

  • programmatic interface
  • click interface (x vs. y)
  • do we need a box-drag interface?  not clear: maybe to edit analysis, persist analysis.   but maybe can push some of this power into the plot.

Chains of analyses requirements

  • Use separate processes for 1  event in a reddis-like decentralized environment?
  • support an arbitrary loopless graph of analysis
  • support for reading/writing EPICS variables in real-time (having analysis/daq/controls be in the same python environment is necessary for this). would help with feedback
  • maybe could replace hutch python to control DAQ with this? (scans)
  • 1 event 1 core
  • scientists only write algorithm boxes
  • two levels: "box" level, and then the "scheduler"
  • how do we pass meta-data through the tree?  (e.g. for two ROIs in a row, the second one needs to know about the first one)
  • policy for type-checking and shape-checking?
  • each analysis box could have optional "output box" for distribution of results

Multiprocessing

  • display makes requests to clients? both algorithm/parameter changes and gather messages
  • different rates for different gather requests?
  • how do you guarantee consistency with changing calculations e.g. ROI?

Reduce/Gather Design Ideas

  • specify what to send, when to send (upstream)
  • what to accept what to do with it
  • send on timer or event-count
  • reduce-last-hundred-events will be fuzzy (alternative is to constrain the way daq hands out events to cores)

Plot "Boxes" Ideas

  • keep them separate from the graphs
  • many output boxes (plot, save to disk, etc.)
  • three big ideas: management, graph, output
  • could be epics

Reconfiguration of Graph

  • reconfigure everything downstream of the box that gets changed
  • boxes like a "deque" box will clear their deque's.
  • boxes could have a variety of more complex reconfigures that have fewer side-effects (e.g. not clear the deques in some circumstances)

Possible Graph Backward Loops

  • This is the case where a background is computed by a post-reduce box and fed to an upstream box
  • We think we can avoid backward loops in this case by having one chain of post-gather boxes feed upstream boxes

Epics

  • Feel like it should be split out of the DAQ timestamped data
  • recorded separately to hdf5 and handled by offline event builder
  • for shmem another process would put epics in some separate buffer (like "epicsStore") for use by AMI
  • could be used to send ami data elsewhere (from output boxes)

Data Interface

  • boxes accept "datagrams" as input/output
  • datagrams contain: evt ids (plural, must be >0), ami-graph-config number, metadata dictionary which has serializable python objects, data dictionary of numpy arrays

IOC Recorders

  • hope these don't need to be processed by AMI, unless DAQ manages to bring them into the hutch-DAQ via "PGP broadcast"
  • if they produce files like current system, AMI would handle them with psana offline event builder
  • as alternative to PGP, online event-builder could grab well-timestamped epics data for us in AMI

DAQ/DRP Reconfig

  • we will assume the DAQ/DRP reconfigs look identical to us
  • DRP reconfigs can be generated from AMI
  • we think DAQ/DRP reconfigs don't interfere with AMI-analysis-graph reconfigs

Display Images at Low Rate

  • no "requests" to graph from display clients
  • use "reduce" boxes

Monitoring of AMI Performance/Health

  • stats sent when data flowing
  • isAlive-health monitored with procserv like process
  • ideally could trigger a python-strace when stuck
  • stats do not flow through the data graph:  separate path (a separate stats piece of the graph?)

DAQ Transitions

  • Try to eliminate transitions in the DAQ
  • instead use "base-configuration" for a run, and time-ranges-of-validity for configuration changes (scans)
  • online and offline event-builders will event-build the configuration changes accordingly

Output Boxes:

  • could be EPICS? (and others)
  • slow data (e.g. slowly changing background) can be posted back to earlier parts in the graph in a slow "database" fashion

Posting Data to Other Calculations

  • All data can be posted
  • Slow data (e.g. slowly changing background) can be posted back to earlier parts of the graph

Names

  • Each box output gets a "name" that users can use to access it for further computation
  • bigger boxes encompassing more steps mean fewer names

pyDM

  • could be useful for assembling AMI windows
  • AMI is made up of relatively few "building block" windows
  • could also be useful for standard-config

To Discuss

  • changing data as run goes on (e.g. DRP pre scaled uncompressed events)

Packages to Consider

dasc

pyqtgraph

mpi

zmq

karabo:
send mail to k. weger

 

vistrails.org:
matplotlib builtin
awkward
python
no obvious c/c++ code
necessary to visualize the workflow
can we group operations into a hierarchy
only qt4 python2 (no python 3)

 

xicam:

http://www.camera.lbl.gov/features-xi-cam

looks interesting.  will talk to LBL developers.

 

orange:

http://orange.biolab.si

looks interesting

recommended as possibility by Tassone/Lenson

 

gasnet.lbl.gov:

data transport used by software like legion

support infiniband


http://vispy.org
clemens: going slowly, recommends pyqtgraph
more for rendering with opengl
replacement for pyqtgraph

https://pypi.python.org/pypi/guidata
user-interface generation. unlikely, but maybe...
www.paraview.org:
no
c++/qt
vtk

 

https://wci.llnl.gov/simulation/computer-codes/visit:
c++, rendering, doesn't look too fancy
vtk

 

http://bokeh.pydata.org

interactivity with 2D visualization not so good?

browser based

 

taverna:
written in java

kepler:
written in java

vtk:
3D visualization? Not so useful

 

http://nerds.airbnb.com/airflow/
workflow (hierarchical batch job) management in python

 

https://developers.google.com/protocol-buffers/
language-independent structured data. doesn't feel too useful.
epicsv4 structured channels:
more wire-protocol stuff

 

http://www.jmp.com/en_us/home.html
plotting
closed source

 

https://www.tableau.com
closed source
software to make nice dashboards

 

nano-surveyor (from camera people)

  • No labels