...

• oriented on XTC processing, but could be extended to work with HDF5 data
• should be easy to use and extend for end users
• support re-use of the existing analysis code
• allow parallel processing on multi-core systems
• common simple configuration of user analysis code

Framework

...

Composition

The centerpiece of the framework is a regular Python application (pyana) which can load one or more user analysis modules which are also written in Python. The core application is responsible for the following tasks:

• loading and initializing all user modules
• reading XTC data from a list of input files
• calling appropriate user methods based on the data being processed
• providing data access to user modules
• providing other services such as histogramming to user modules

...

The core application have a number of configuration options which can be set or changed from configuration file or from a command line. If the same option appears both in configuration file and command line the n the command line value overrides value in the configuration file.

User Modules

User analysis module is a regular Python module (a Python file) which satisfies additional requirements:

...

The application loads one or more user modules, the names of the modules to load are specified either in the job configuration file or on the command line. After loading the modules the application creates one or more instance objects of the class defined in the module. More than one instance may be useful if one wants to run the same analysis with different set of parameters in the same job. Number of instances and their parameters are determined by the job configuration file (see below).

Initialization

User analysis class can define zero or more parameters in its constructor (__init__() method). Parameters are initialized from the values defined in the job configuration file (see below). All parameters are passed to the Python code as strings, if the code expects a number or some other type then it's the code responsibility to convert the strings to appropriate type. If there are no default values defined for some parameters in constructor declaration then those parameters must be present in the configuration file.

...

(the order of parameters in constructor and configuration file does not matter as all parameters are passed as keyword parameters.)

Data access

There are two types of data that framework passes to the user analysis modules – event data and environment data. Event data contains the data corresponding to current event that triggered the call to the user methods. In case of XTC input the event data contains complete datagram as read from DAQ. Event data in user module is represented with a special object of type pyana.event.Event which has an extended interface for extracting individual object from datagram. This interface is described in the reference guide.

Environment data include all kinds of data which are not part of the event data. Usually environment data either stay the same for the whole job or change at a slower rate than event data. Example of the environment data could be configuration data read from XTC at the beginning of the job, EPICS data which is not updated on every event, and few other things. Environment data is represented for user code through the object of type pyana.event.Env. Its interface is described in the reference guide.

Data Source Address

For some pieces of data one needs to specify data "address" which identifies (maybe partially) particular DAQ device which produced the data. This is needed because the instrument setup may include multiple devices producing the same data type. The DAQ defines a type which serves as a most specific device identification, the type is xtc.DetInfo in package pypdsdata. One can pass this DetInfo instance to a method which accepts device address to select that specific device. DetInfo object contains four essential pieces of information:

...

• "AmoETof-0|Acqiris-0" – selects data produced by detector AmoETof, detId 0, device Acqiris, devId 0
• "AmoETof|Acqiris" – selects data produced by detector AmoETof, any detId, device Acqiris, any devId
• "AmoETof-*|Acqiris-*" – same as above
• "AmoETof-0" – selects data produced by detector AmoETof, detId 0, any device, any devId
• "|Acqiris-0" – selects data produced by any detector, any detId, device Acqiris, devId 0
• "*-*|Acqiris-0" – same as above

Configuration

Analysis job can read the configuration options from the command line and/or the configuration file. Command line can be used to set options only for the pyana application itself but not user analysis modules. Options for user modules can be set in configuration file only.

Configuration File

The default name for the configuration file is "pyana.cfg" in the current directory. The name can be changed with the --config or -c command line option. The format of the configuration file follows the widely accepted syntax for INI files. File consists of series of sections, beginning of each section is marked by the section name in square brackets. Each section contains arbitrary number of options in the form option = value. Both section header and option name must start on first column, white space at the beginning of the line means that line is a continuation of the previous option line. Empty lines and lines beginning with word # (hash sign) or ; (semicolon) are considered comments. Semicolon anywhere on the line as considered a beginning of the comment if it follows space character. The word REM (case insensitive) at the beginning of the line starts a comment if it is followed by space, TAB or new-line.

Here is an example of the configuration file syntax:

# Comment line
; Another comment
[section1]
name = Default    ; In-line comment

# next line shows line continuation
modules = module.A module.B
          module.C

REM REM-style comment
[section2]
limit = 1000

Core Options

By default the core application options are read from {{\[pyana\]}} section of the configuration file. If the option {{-C _name_}} or {{--config-name=_name_}} is given on the command line then additional section {{\[pyana._name_]\}} is read and values in that section override values from {{\[pyana\]}} section.

Here is the list of all command line and configuration file options availabale currently:

Multi-processing

The framework can be run in single-process or multi-process mode, by default everything runs in singe-process mode. In multi-process mode analysis job spawns a number of processes all running on the same host. In that case framework is also responsible for distributing individual events to a single or multiple processes and collecting and merging the results of the processing at the end of the job.