Introduction

This document describes C++ analysis framework for LCLS and how users can make use of its features. Psana design borrows ideas from multitude of other framworks such as pyana, myana, BaBar framework, etc. It's main principles are summarized here:

• support processing of both XTC and HDF5 data format
• user code should be independent of specific data format
• should be easy to use and extend for end users
• support re-use of the existing analysis code
• common simple configuration of user analysis code

This manual is accompanied by the psana - Reference Manual which describes interfaces of the classes available in Psana.

Framework Architecture

The central part of the framework is a regular pre-built application (psana) which can dynamically load one or more user analysis modules which are written in C++. The core application is responsible for the following tasks:

• loading and initializing all user modules
• loading one of the input modules to read data from XTC or HDF5
• calling appropriate methods of user modules based on the data being processed
• providing access to data as set of C++ classes
• providing other services such as histogramming to user modules

Other important components of the Psana architecture:

• user module – instance of the C++ class which inherits pre-defined Module class and defines few special methods which are called by framework
• event – special object which transparently stores all event data
• environment – special object which stores non-event data such as configuration objects or EPICS data

Analysis Job Life Cycle

Psana analysis job goes through cycles of state changes such as initialization, configuration, event processing, etc. calling methods of the user modules at every such change. This model follows closely the production activities in LCLS on-line system. DAQ system defines many types of transitions in its data-taking activity, most interesting are here:

• Configure - provides configuration data for complete setup
• BeginRun - start of data taking for one run
• BeginCalibCycle - start of the new scan, some configuration data may change at his point
• L1Accept - this is regular event containing event data from all detectors
• EndCalibCycle - end of single scan
• EndRun - end of data taking for one run
• Unconfigure - stop of all activity

Typically there will be more than one run taken with the same configuration, so there may be more than one BeginRun/EndRun transition for one Configure/Unconfigure, but a data file from single run should contain only one BeginRun/EndRun. Depending on a setup there could be one or more BeginCalibCycle/EndCalibCycle transitions in single run.

For each of the above transitions psana will call corresponding method in user modules notifying them of the possible change in the configuration or just providing event data. Following method names are defined in the user modules:

• beginJob() – this method is called once per analysis job when first Configure transition happens. If there is more than one Configure in single job (when processing multiple runs) this method is not called, use beginRun() to observe configuration changes in this case. This method can access all configuration data through environment object.
• beginRun() – this method is called for every new BeginRun, so it will be called multiple times when processing multiple runs in the same job. This method can access all configuration data through environment object.
• beginCalibCycle() – this method is called for every new BeginCalibCycle, so it will be called multiple times when processing multiple runs in the same job or when single run contains multiple scans. This method can access all configuration data through environment object.
• event() – this method is called for every new L1Accept, it has access to event data through event object as well as configuration data through environment object.
• endCalibCycle() – this method is called for every new EndCalibCycle, it has access to configuration data through environment object.
• endRun() – this method is called for every new EndRun, it has access to configuration data through environment object.
• endJob() – this method is called once at the end of analysis job, it has access to configuration data through environment object.

Typically psana will iterate through all transitions/events from the input files. User modules have a limited control over this event loop, module can request to skip particular event, stop iteration early or abort job using one of the methods described below.

User Modules

User module in psana is an instance of the C++ class which inherits from the Module class (defined in file pasana/Module.h) and implements several methods. These methods are already mentioned above, here is more formal description of each method:

• void beginJob(Event& evt, Env& env)
  Method called once at the beginning of the job. Environment object contains configuration data from the first Configure transition. Default implementation of this method does not do anything.
• void beginRun(Event& evt, Env& env)
  Method called at the beginning of every new run. Default implementation of this method does not do anything.
• void beginCalibCycle(Event& evt, Env& env)
  Method called at the beginning of every new scan. Default implementation of this method does not do anything.
• void event(Event& evt, Env& env)
  Method called for every regular event. Even data is accessible through =evt= argument. There is no default implementation for this method and user module must provide at least this method.
• void endCalibCycle(Event& evt, Env& env)
  Method called at the end of every new scan, can be used to process scan-level statistics collected in event(). Default implementation of this method does not do anything.
• void endRun(Event& evt, Env& env)
  Method called at the end of every run, can be used to process run-level statistics collected in event(). Default implementation of this method does not do anything.
• void endJob(Event& evt, Env& env)
  Method called once at the end of analysis job, can be used to process job-level statistics collected in event(). Default implementation of this method does not do anything.

In addition to event() method every module class must provide a constructor which takes string argument giving the name of the module. Additionally it has to provide a special factory function use to instantiate the modules from the shared libraries, there is special macro defined for definition of this factory function.

Here is the minimal example of the module class declaration with only the event() method implemented and many non-essential details are skipped:

{include:PageMenuBegin}
{toc}
{include:PageMenuEnd}

h1. Introduction

This document describes C+\+ analysis framework for LCLS and how users can make use of its features. Psana design borrows ideas from multitude of other framworks such as pyana, myana, BaBar framework, etc. It's main principles are summarized here:

* support processing of both XTC and HDF5 data format
* user code should be independent of specific data format
* should be easy to use and extend for end users
* support re-use of the existing analysis code
* common simple configuration of user analysis code

This manual is accompanied by the [psana - Reference Manual] which describes interfaces of the classes available in Psana.





h1. Framework Architecture

The central part of the framework is a regular pre-built application (psana) which can dynamically load one or more user analysis modules which are written in C++. The core application is responsible for the following tasks:

* loading and initializing all user modules
* loading one of the input modules to read data from XTC or HDF5
* calling appropriate methods of user modules based on the data being processed
* providing access to data as set of C+\+ classes
* providing other services such as histogramming to user modules

Other important components of the Psana architecture:

* _user module_ -- instance of the C+\+ class which inherits pre-defined Module class and defines few special methods which are called by framework
* _event_ -- special object which transparently stores all event data
* _environment_ -- special object which stores non-event data such as configuration objects or EPICS data







h1. Analysis Job Life Cycle

Psana analysis job goes through cycles of state changes such as initialization, configuration, event processing, etc. calling methods of the user modules at every such change. This model follows closely the production activities in LCLS on-line system. DAQ system defines many types of transitions in its data-taking activity, most interesting are here:
* {{Configure}} \- provides configuration data for complete setup
* {{BeginRun}} \- start of data taking for one run
* {{BeginCalibCycle}} \- start of the new scan, some configuration data may change at his point
* {{L1Accept}} \- this is regular event containing event data from all detectors
* {{EndCalibCycle}} \- end of single scan
* {{EndRun}} \- end of data taking for one run
* {{Unconfigure}} \- stop of all activity

Typically there will be more than one run taken with the same configuration, so there may be more than one {{BeginRun/EndRun}} transition for one {{Configure/Unconfigure}}, but a data file from single run should contain only one {{BeginRun/EndRun}}. Depending on a setup there could be one or more {{BeginCalibCycle/EndCalibCycle}} transitions in single run.

For each of the above transitions psana will call corresponding method in user modules notifying them of the possible change in the configuration or just providing event data. Following method names are defined in the user modules:
* *beginJob()* -- this method is called *once* per analysis job when first {{Configure}} transition happens. If there is more than one {{Configure}} in single job (when processing multiple runs) this method is not called, use beginRun() to observe configuration changes in this case. This method can access all configuration data through environment object.
* *beginRun()* -- this method is called for every new {{BeginRun}}, so it will be called multiple times when processing multiple runs in the same job. This method can access all configuration data through environment object.
* *beginCalibCycle()* -- this method is called for every new {{BeginCalibCycle}}, so it will be called multiple times when processing multiple runs in the same job or when single run contains multiple scans. This method can access all configuration data through environment object.
* *event()* -- this method is called for every new {{L1Accept}}, it has access to event data through event object as well as configuration data through environment object.
* *endCalibCycle()* -- this method is called for every new {{EndCalibCycle}}, it has access to configuration data through environment object.
* *endRun()* -- this method is called for every new {{EndRun}}, it has access to configuration data through environment object.
* *endJob()* -- this method is called *once* at the end of analysis job, it has access to configuration data through environment object.

Typically psana will iterate through all transitions/events from the input files. User modules have a limited control over this event loop, module can request to skip particular event, stop iteration early or abort job using one of the methods described below.





h1. User Modules

User module in psana is an instance of the C+\+ class which inherits from the {{Module}} class (defined in file {{pasana/Module.h}}) and implements several methods. These methods are already mentioned above, here is more formal description of each method:

* {{void}} {{{*}beginJob{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called once at the beginning of the job. Environment object contains configuration data from the first {{Configure}} transition. Default implementation of this method does not do anything.
* {{void}} {{{*}beginRun{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called at the beginning of every new run. Default implementation of this method does not do anything.
* {{void}} {{{*}beginCalibCycle{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called at the beginning of every new scan. Default implementation of this method does not do anything.
* {{void}} {{{*}event{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called for every regular event. Even data is accessible through =evt= argument. There is no default implementation for this method and user module must provide at least this method.
* {{void}} {{{*}endCalibCycle{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called at the end of every new scan, can be used to process scan-level statistics collected in {{event()}}. Default implementation of this method does not do anything.
* {{void}} {{{*}endRun{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called at the end of every run, can be used to process run-level statistics collected in {{event()}}. Default implementation of this method does not do anything.
* {{void}} {{{*}endJob{*}{}}}{{(Event& evt, Env& env)}}
\\
Method called once at the end of analysis job, can be used to process job-level statistics collected in {{event()}}. Default implementation of this method does not do anything.

In addition to {{event()}} method every module class must provide a constructor which takes string argument giving the name of the module. Additionally it has to provide a special factory function use to instantiate the modules from the shared libraries, there is special macro defined for definition of this factory function.

Here is the minimal example of the module class declaration with only the {{event()}} method implemented and many non-essential details are skipped:

{code:title=Package/ExampleModule.h|borderStyle=solid}
#include "psana/Module.h"

namespace Package {
class ExampleModule: public Module {
public:

  // Constructor takes module name as a parameter
  ExampleModule(const std::string& name);

  // Implementation of event() from base class
  virtual void event(Event& evt, Env& env);

};
} // namespace Package
{code}

Definition

...

of

...

the

...

factory

...

function

...

and

...

methods:

}
#include "Package/ExampleModule.h"
#include "MsgLogger/MsgLogger.h"
#include "PSEvt/EventId.h"

// define factory function
using namespace Package;
PSANA_MODULE_FACTORY(ExampleModule)

// Constructor
ExampleModule::ExampleModule(const std::string& name)
  : Module(name)
{
}

void
ExampleModule::event(Event& evt, Env& env)
{
  // get event ID
  shared_ptr<EventId> eventId = evt.get();
  if (not eventId.get()) {
    MsgLog(name(), info, "event ID not found");
  } else {
    MsgLog(name(), info, "event ID: " << *eventId);
  }
}
{code}

This

...

simple

...

example

...

already

...

does

...

something

...

useful,

...

it

...

retrieves

...

and

...

prints

...

event

...

ID

...

(copied

...

from

...

standard

...

PrintEventId

...

module).

...

Actual

...

modules

...

will

...

do

...

more

...

complex

...

things

...

but

...

this

...

is

...

a

...

simple

...

example

...

of

...

obtaining

...

something

...

from

...

event

...

data.

...

The

...

easiest

...

way

...

to

...

write

...

new

...

user

...

modules

...

is

...

to

...

use

...

codegen

...

script

...

to

...

generate

...

class

...

from

...

predefined

...

template.

...

This

...

command

...

will

...

create

...

new

...

module

...

ExampleModule

...

in

...

package

...

TestPackage

...

and

...

will

...

copy

...

generated

...

files

...

to

...

the

...

directories

...

in

...

TestPackage:

}
codegen -l psana-module TestPackage ExampleModule
{code}

h1. Controlling Framework from User Module


Code in user modules can control framework event loop by calling one of the three methods:

* {{void skip()}}
\\
Signal framework to skip current event and do not call other downstream modules. Note that this method does not skip code in the current module, control is returned back to the module. If you want to stop processing after this call then add a return statement.
* {{void stop()}}
\\
Signal framework to stop event loop and finish job gracefully (with calling 

Controlling Framework from User Module

Code in user modules can control framework event loop by calling one of the three methods:

• void skip()
  Signal framework to skip current event and do not call other downstream modules. Note that this method does not skip code in the current module, control is returned back to the module. If you want to stop processing after this call then add a return statement.
• void stop()
  Signal framework to stop event loop and finish job gracefully (with calling endRun/endJob/etc.).

...

• Note

...

• that

...

• this

...

• method

...

• does

...

• not

...

• terminate

...

• processing

...

• in

...

• the

...

• current

...

• module.

...

• If

...

• you

...

• want

...

• to

...

• stop

...

• processing

...

• after

...

• this

...

• call

...

• then

...

• add

...

• a

...

• return

...

• statement.

...

• void

...

• terminate()
  Signal framework to terminate immediately. Note that this method does not terminate processing in the current module. If you want to stop processing after this call then add a return statement.

Here is an example of the code using above functions:

}}
\\
Signal framework to terminate immediately. Note that this method does not terminate processing in the current module. If you want to stop processing after this call then add a return statement.

Here is an example of the code using above functions:
{code}
void ExampleModule::event(Event& evt, Env& env) {

  ...

  if (pixelsAboveThreshold < 1000) {
    // This event is not worth looking at, skip it
    skip();
    // I do not want to continue with this algorithm either
    return;
  }

  if (nGoodEvents > 1000) {
    // we collected enough data, can stop now and go to endJob()
    stop();
    // I do not want to continue with this algorithm either
    return;
  }

  if (temperatureKelvin < 0) {
    // data is junk, stop right here and don't call endJob()
    terminate();
    // I do not want to continue with this algorithm either
    return;
  }

}
{code}

Skipped

...

events

...

can

...

be

...

used

...

in

...

further

...

analysis

...

or

...

saved

...

in

...

the

...

"filtered"

...

Xtc

...

file,

...

as

...

explained

...

in

...

Package PSXtcOutput.

Job and Module Configuration

Psana framework has multiple configuration parameters that can be changed via command line or special configuration file. Configuration file can also specify parameters for user modules so that modules' behavior can be changed at run time without the need to recompile the code.

If no options are specified on the command line then psana tries to read configuration file named psana.cfg from the current directory if that file exists. The location of the configuration file can be changed with the -c <path> option which should provide path of the configuration file.

Configuration File Format

Configuration file has a simple format which is similar to well-known INI file format. The file consists of the sections, each section begins with the section header in the form:

 PSXtcOutput|PCDS:Psana Reference Manual#Package PSXtcOutput].




h1. Job and Module Configuration


Psana framework has multiple configuration parameters that can be changed via command line or special configuration file. Configuration file can also specify parameters for user modules so that modules' behavior can be changed at run time without the need to recompile the code.

If no options are specified on the command line then psana tries to read configuration file named {{psana.cfg}} from the current directory if that file exists. The location of the configuration file can be changed with the {{\-c <path>}} option which should provide path of the configuration file.



h2. Configuration File Format

Configuration file has a simple format which is similar to well-known {{INI file format}}. The file consists of the sections, each section begins with the section header in the form:
{code}
[<section-name>]
{code}

Section

...

names

...

can

...

be

...

arbitrary

...

strings,

...

but

...

in

...

psana

...

case

...

section

...

names

...

are

...

the

...

names

...

of

...

the

...

modules

...

which

...

cannot

...

be

...

arbitrary

...

and

...

should

...

not

...

contain

...

spaces.

...

Following

...

the

...

section

...

header

...

there

...

may

...

be

...

zero

...

or

...

more

...

parameter

...

lines

...

in

...

the

...

form

}
<param-name> = <param-value>
{code}

Parameter

...

name

...

is

...

anything

...

between

...

beginning

...

of

...

line

...

and

...

'='

...

character

...

with

...

leading

...

and

...

trailing

...

spaces

...

and

...

tabs

...

stripped.

...

Parameter

...

value

...

is

...

anything

...

after

...

'='

...

character

...

with

...

leading

...

and

...

trailing

...

spaces

...

and

...

tabs

...

stripped,

...

parameter

...

value

...

can

...

be

...

empty.

...

Long

...

parameter

...

value

...

can

...

be

...

split

...

over

...

multiple

...

lines

...

if

...

the

...

line

...

ends

...

with

...

the

...

backslash

...

character,

...

e.g.:

}
files = /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s00-c00.xtc \
        /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s01-c00.xtc \
        /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s02-c00.xtc
{code}

Lines

...

starting

...

with

...

'#'

...

character

...

are

...

considered

...

comments

...

and

...

ignored.

Parameter Types

Configuration file does not specify parameter types, all values in the file are strings. Psana framework provides conversion of these strings to several basic C++ types or sequences. Following types and conversion rules are supported by framework:

• bool
  value strings "yes", "true", "on" become true, "no", "false", "off" become false. Strings which represent non-zero numbers become true, string "0" becomes false.
• char
  value string must be single-character string and it will be assigned to a result.
• C++ numeric types
  option value must represent valid number.
• std::string
  option value will be assigned to result string without change.
• C++ sequence types (e.g. std::list<T>)
  option value will be split into single words at space/tab characters, individual words will be converted to resulting type T.

When the conversion fails because of the incorrectly formatted input framework will throw an exception with the type ExceptionCvtFail.

Psana Parameters

The parameters that are needed for the framework are defined in psana section. Here is the list of parameters which can appear in that section:

• modules
  list of module names to include in the analysis job. Each module name is built of a package name and class name separated by dot (e.g. TestPackage.ExampleModule) optionally followed by colon and modifier. Modifier is not needed if there is only one instance of the module in the job. If there is more than on instance then modules need to include unique modifier to distinguish instances. If the module comes from psana package then package name can be omitted. Module names can also be specified on the command line with -m option, for multiple modules use multiple -m options or comma-separated names in single -m option.
• files
  list of file names to process. File names can also be specified on the command line which will override anything specified in configuration file.
• events
  maximum number of events to process in a job, can also be given on the commnad line with -n or --num-events option.
• skip-events
  number of events to skip before starting even processing, can also be given on the commnad line with -s or --skip-events option.
• instrument
  Instrument name.
• experiment
  Experiment name. Instrument and expriment names can be specified on the commnad line with -e or --experiment option, option value has format XPP:xpp12311 or xpp12311. By default instrument and experiment names are determined from input file names, you can use these options to override defaults (or when your file has non-standard naming).
• calib-dir
  Path to the calibration directory, can also be given on the commnad line with -b or --calib-dir option. Path can include {instr} and {exp} strings which will be replaced with instrument and experiment names respectively. Default value for path is /reg/d/psdm/{instr}/{exp}/calib.

Here is an example of the framework configuration section:

h2. Parameter Types

Configuration file does not specify parameter types, all values in the file are strings. Psana framework provides conversion of these strings to several basic C+\+ types or sequences. Following types and conversion rules are supported by framework:
* {{bool}}
\\
value strings "yes", "true", "on" become {{true}}, "no", "false", "off" become {{false}}. Strings which represent non-zero numbers become {{true}}, string "0" becomes {{false}}.
* {{char}}
\\
value string must be single-character string and it will be assigned to a result.
* C+\+ numeric types
\\
option value must represent valid number.
* {{std::string}}
\\
option value will be assigned to result string without change.
* C+\+ sequence types (e.g. {{std::list<T>}})
\\
option value will be split into single words at space/tab characters, individual words will be converted to resulting type {{T}}.

When the conversion fails because of the incorrectly formatted input framework will throw an exception with the type {{ExceptionCvtFail}}.



h2. Psana Parameters

The parameters that are needed for the framework are defined in {{[psana]}} section. Here is the list of parameters which can appear in that section:
* {{modules}}
\\
list of module names to include in the analysis job. Each module name is built of a package name and class name separated by dot (e.g. {{TestPackage.ExampleModule}}) optionally followed by colon and modifier. Modifier is not needed if there is only one instance of the module in the job. If there is more than on instance then modules need to include unique modifier to distinguish instances. If the module comes from psana package then package name can be omitted. Module names can also be specified on the command line with {{\-m}} option, for multiple modules use multiple {{\-m}} options or comma-separated names in single \-m option.
* {{files}}
\\
list of file names to process. File names can also be specified on the command line which will override anything specified in configuration file.
* {{events}}
\\
maximum number of events to process in a job, can also be given on the commnad line with {{\-n}} or {{\--num-events}} option.
* {{skip-events}}
\\
number of events to skip before starting even processing, can also be given on the commnad line with {{\-s}} or {{\--skip-events}} option.
* {{instrument}}
\\
Instrument name.
* {{experiment}}
\\
Experiment name. Instrument and expriment names can be specified on the commnad line with {{\-e}} or {{\--experiment}} option, option value has format {{XPP:xpp12311}} or {{xpp12311}}. By default instrument and experiment names are determined from input file names, you can use these options to override defaults (or when your file has non-standard naming).
* {{calib-dir}}
\\
Path to the calibration directory, can also be given on the commnad line with {{\-b}} or {{\--calib-dir}} option. Path can include {{\{instr\}}} and {{\{exp\}}} strings which will be replaced with instrument and experiment names respectively. Default value for path is {{/reg/d/psdm/\{instr\}/\{exp\}/calib}}.

Here is an example of the framework configuration section:
{code}
[psana]
# list of file names
files = /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s00-c00.xtc \
        /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s01-c00.xtc \
        /reg/d/psdm/AMO/amo00000/xtc/e00-r0000-s02-c00.xtc
# list of modules, PrintSeparator and PrintEventId are from psana package
# and do not need package name
modules = PrintSeparator PrintEventId psana_examples.DumpAcqiris
{code}



h2. User Modules Parameters

Parameters for user modules appear in the separate sections named after the modules. For example the module with name 

User Modules Parameters

Parameters for user modules appear in the separate sections named after the modules. For example the module with name "TestPackage.ExampleModule"

...

will

...

read

...

its

...

parameters

...

from

...

the

...

section

...

[TestPackage.ExampleModule

...

]

...

.

...

If

...

the

...

module

...

name

...

includes

...

modifier

...

after

...

colon

...

then

...

it

...

will

...

try

...

to

...

find

...

parameter

...

value

...

in

...

the

...

corresponding

...

section

...

first

...

and

...

if

...

it

...

does

...

not

...

exist

...

there

...

it

...

will

...

try

...

to

...

read

...

parameter

...

form

...

section

...

which

...

does

...

not

...

have

...

modifier.

...

In

...

this

...

way

...

the

...

modules

...

can

...

share

...

common

...

parameters.

...

For

...

example

...

the

...

module

...

"TestPackage.ExampleModule:test"

...

will

...

try

...

to

...

read

...

a

...

parameter

...

from

...

[TestPackage.ExampleModule:test

...

]

...

section

...

first

...

and

...

[TestPackage.ExampleModule

...

]

...

section

...

after

...

that.

...

Here

...

is

...

an

...

example

...

of

...

configuration

...

for

...

some

...

fictional

...

analysis

...

job:

}
[psana]
modules = TestPackage.Analysis:mode1 TestPackage.Analysis:mode2

[TestPackage.Analysis]
# these are common parameters for all TestPackage.Analysis modules,
# but instances can override then in their own sections
calib-mode = fancy
subpixel = off
threshold = 0.001

[TestPackage.Analysis:mode1]
# parameters specific to :mode1 module
range-min = 0
range-max = 1000000

[TestPackage.Analysis:mode2]
# parameters specific to :mode2 module
range-min = 1000
range-min = 10000
subpixel = on
{code}



h2. Accessing Configuration Parameters

User module base class defines few convenience methods which simplify access to configuration parameters. Here is the list of the methods:
* {{std::string}} {{{*}configStr{*}{}}}{{(const 

Accessing Configuration Parameters

User module base class defines few convenience methods which simplify access to configuration parameters. Here is the list of the methods:

• std::string configStr(const std::string&

...

• param)

...

• 
  this method takes the name of the parameter and returns full parameter value as a string. If parameter cannot be found the exception will be thrown.
• T config(const std::string&

...

• param)

...

• 
  this method takes the name of the parameter and returns parameter value converted to type T. If parameter cannot be found the exception will be thrown.
• std::string configStr(const std::string&

...

• param,

...

• const

...

• std::string&

...

• def)

...

• 
  this method takes the name of the parameter and returns full parameter value as a string. If parameter cannot be found then the value of second argument will be returned.
• T config(const std::string&

...

• param,

...

• T

...

• def)

...

• 
  this method takes the name of the parameter and returns parameter value converted to type T. If parameter cannot be found then the value of second argument will be returned.
• Seq configList(const std::string&

...

• param)

...

• 
  this method takes the name of the parameter and returns parameter value converted to sequence. Sequence can be any of standard container types such as std::list<std::string>

...

• or

...

• std::vector<double>

...

• .

...

• If

...

• parameter

...

• cannot

...

• be

...

• found

...

• the

...

• exception

...

• will

...

• be

...

• thrown.

...

• std::list<T>

...

• configList(const

...

• std::string&

...

• param,

...

• const

...

• std::list<T>&

...

• def)

...

• 
  this method takes the name of the parameter and returns parameter value converted to std::list<T>

...

• .

...

• If

...

• parameter

...

• cannot

...

• be

...

• found

...

• then

...

• the

...

• value

...

• of

...

• second

...

• argument

...

• will

...

• be

...

• returned.

...

Here

...

is

...

an

...

example

...

of

...

the

...

code

...

in

...

user

...

module

...

which

...

uses

...

these

...

methods:

}
  Source src = configStr("source", "DetInfo(:Evr)");
  int repeat = config("repeat");
  std::list<std::string> options = configList("options");
{code}




h1. Messaging Service

In many cases the user modules want to produce/print messages such as errors, warnings, or debugging information. In most cases C+\+ code uses standard C+\+ facilities such as {{std::cout}}, {{std::cerr}}, or even {{printf}} to format/print something to the terminal or log file. Psana framework provides different approach for messaging which provides better control for the output level (e.g. turning on/off debugging) and better flexibility.

Each message produced by messaging service carries corresponding level. There are several levels of messages defined by the service:
* {{debug}} -- lowest message level reserved for debugging messages, normally turned off during normal running
* {{trace}} -- one level higher than {{debug}}, normally turned off during normal running
* {{info}} -- level for regular informational messages, normally printed but can be turned off
* {{warning}} -- level for warnings which are not errors
* {{error}} -- level for error messages
* {{fatal}} -- level for fatal errors, after the message is published the program will terminate
The levels are ordered, enabling messages of one level also enables messages of all higher levels.

Each logging message is associated with one _logger_. Loggers have names which form hierarchical structure such as 

Messaging Service

In many cases the user modules want to produce/print messages such as errors, warnings, or debugging information. In most cases C++ code uses standard C++ facilities such as std::cout, std::cerr, or even printf to format/print something to the terminal or log file. Psana framework provides different approach for messaging which provides better control for the output level (e.g. turning on/off debugging) and better flexibility.

Each message produced by messaging service carries corresponding level. There are several levels of messages defined by the service:

• debug – lowest message level reserved for debugging messages, normally turned off during normal running
• trace – one level higher than debug, normally turned off during normal running
• info – level for regular informational messages, normally printed but can be turned off
• warning – level for warnings which are not errors
• error – level for error messages
• fatal – level for fatal errors, after the message is published the program will terminate
  The levels are ordered, enabling messages of one level also enables messages of all higher levels.

Each logging message is associated with one logger. Loggers have names which form hierarchical structure such as "GrandParent.Parent.Child".

...

Top-level

...

logger

...

has

...

no

...

name

...

and

...

is

...

called

...

root

...

logger

...

.

...

Loggers

...

were

...

introduced

...

for

...

flexibility,

...

it

...

is

...

possible

...

to

...

configure

...

individual

...

loggers,

...

for

...

example

...

to

...

enable

...

debug

...

logging

...

from

...

one

...

particular

...

logger.

...

Good

...

practice

...

is

...

to

...

use

...

logger

...

name

...

which

...

is

...

the

...

same

...

as

...

user

...

module

...

name

...

for

...

identification

...

purposes.

...

To

...

use

...

messaging

...

service

...

one

...

has

...

to

...

include

...

header

...

file

...

"MsgLogger/MsgLogger.h"

...

which

...

defines

...

a

...

set

...

of

...

macros

...

for

...

message

...

logging

...

and

...

all

...

related

...

classes.

...

User

...

code

...

interacts

...

with

...

the

...

messaging

...

service

...

through

...

this

...

set

...

of

...

macros:

...

• MsgLog(logger,

...

• level,

...

• message)

...

• //

...

• send

...

• a

...

• message

...

• to

...

• specific

...

• logger,

...

• takes

...

• logger

...

• name,

...

• logging

...

• level,

...

• and

...

• message.

...

• Message

...

• is

...

• a

...

• construct

...

• which

...

• can

...

• appear

...

• after

...

• stream

...

• insertion

...

• operator

...

• (e.g.

...

• cout

...

• <<

...

• message

...

• ).

...

• MsgLogRoot(level,

...

• message)

...

• //

...

• same

...

• as

...

• above

...

• but

...

• message

...

• is

...

• sent

...

• to

...

• root

...

• logger.

...

Here

...

are

...

few

...

examples

...

of

...

using

...

these

...

macros:

}
  MsgLog("MyModule", info, "reading pedestals from file " << fileName);
  MsgLog("MyModule", debug, "intermediate result: count=" << count << " sum=" << sum);
  MsgLogRoot(warning, "warp engine overheating");
{code}

Note:

...

in

...

user

...

module

...

replace

...

"MyModule"

...

string

...

with

...

the

...

name()

...

call

...

which

...

returns

...

the

...

name

...

of

...

the

...

user

...

module.

...

Above

...

macros

...

are

...

simple

...

to

...

use

...

in

...

most

...

cases

...

as

...

they

...

hide

...

all

...

details

...

from

...

user.

...

In

...

more

...

complex

...

situations

...

(printing

...

array

...

elements)

...

there

...

are

...

two

...

macros

...

which

...

provide

...

access

...

to

...

underlying

...

stream

...

object

...

which

...

can

...

be

...

used

...

in

...

more

...

interesting

...

ways:

• 
  this macro declares stream object which can be used by the code in compound statement which follows the macro. The lifetime of the stream is the code block, after the code block is executed the message is published and stream disappears.
• 
  variation of the above macro which publishes message to root logger.

Here is an example of their use:

* {html}<TT>WithMsgLog(logger, level, stream) { ... }</TT>{html}
\\
this macro declares stream object which can be used by the code in compound statement which follows the macro. The lifetime of the stream is the code block, after the code block is executed the message is published and stream disappears.
* {html}<TT>WithMsgLogRoot(level, stream) { ... }</TT>{html}
\\
variation of the above macro which publishes message to root logger.

Here is an example of their use:
{code}
  WithMsgLog("MyModule", debug, str) {
    str << "array elements:";
    for (int i = 0; i < size; ++ i) {
      str << " " << array[i];
    }
  }
{code}

When

...

messaging

...

service

...

delivers

...

(prints)

...

the

...

message

...

in

...

addition

...

to

...

message

...

itself

...

it

...

provides

...

additional

...

information

...

about

...

message.

...

In

...

psana

...

it

...

will

...

print

...

level

...

name

...

and

...

logger

...

name;

...

for

...

trace

...

messages

...

it

...

will

...

also

...

print

...

timestamp;

...

for

...

debug

...

and

...

error

...

messages

...

it

...

will

...

print

...

timestamp

...

and

...

location

...

(file

...

name

...

and

...

line

...

number)

...

where

...

message

...

originated.

...

By

...

default

...

psana

...

enables

...

messages

...

of

...

the

...

info

...

level

...

(and

...

higher).

...

To

...

enable

...

lower

...

level

...

messages

...

one

...

can

...

provide

...

-v

...

option

...

to

...

psana:

...

one

...

-v

...

will

...

enable

...

trace

...

messages,

...

two

...

-v

...

options

...

will

...

enable

...

debug

...

messages.

...

To

...

disable

...

info

...

and

...

warning

...

messages

...

one

...

can

...

provide

...

one

...

or

...

two

...

-q

...

options.

...

Error

...

and

...

fatal

...

messages

...

cannot

...

be

...

disabled.

...

Note:

...

when

...

the

...

message

...

level

...

is

...

disabled

...

the

...

code

...

in

...

the

...

corresponding

...

macros

...

is

...

not

...

executed

...

at

...

all.

...

Do

...

not

...

put

...

any

...

expressions

...

with

...

side

...

effects

...

into

...

message

...

or

...

code

...

blocks,

...

these

...

are

...

strictly

...

for

...

messaging,

...

not

...

part

...

of

...

your

...

algorithm.

Histogramming Service

Psana includes a histogramming service which is wrapper for ROOT histogramming package. This service simplifies several tasks such as opening ROOT file, saving histograms to file, etc.

Center piece of the histogramming service is the histogram manager class. Histogram manager's responsibilities is to open ROOT file, create histograms, and to store histograms to the file. All these tasks are performed transparently to user, there is no need for additional configuration of this service. To create histograms one needs first to obtain a reference to a manager instance which is a part of the standard psana environment and is accessible through a method of the environment class. One then can call factory methods of the manager class to create new histograms which will be automatically saved to a ROOT file. The manager creates a single ROOT file to store all histograms created in a single job. Then name of the ROOT file is the same as the job name with ".root" extension added. The name of psana job is auto-generated from the name of the first input file, but it can also be set on the command line with -j <job-name> option.

All factory methods of the histogram manager use special class to describe histogram axis (or axes for 2-dim histograms). The name of the class is PSHist::Axis (in the user module PSHist:: prefix is optional) and it contains binning information for single histogram axis. It can be constructed in two different ways:

• Axis(int nbins, double amin, double amax)
  defines axis with fixed-width bins in the range from amin to amax.
• Axis(int nbins, const double* edges)
  defines axis with variable-width bins, array contains the low edge of each bin plus high edge of the last bin. Total size of the edges array must be nbins+1.

Here is the list of the factory methods (see also reference for more information):

• PSHist::H1* hist1i(const std::string& name, const std::string& title, const Axis& axis)
  creates one-dimensional histogram with integer bin contents. Returns pointer to histogram object.
• PSHist::H1* hist1d(name, title, axis)
  (argument types same as above) creates one-dimensional histogram with double (64-bit) bin contents. Returns pointer to histogram object.
• PSHist::H1* hist1f(name, title, axis)
  creates one-dimensional histogram with float (32-bit) bin contents. Returns pointer to histogram object.
• PSHist::H2* hist2i(name, title, xaxis, yaxis)
  creates two-dimensional histogram with integer bin contents. Returns pointer to histogram object.
• PSHist::H2* hist2d(name, title, xaxis, yaxis)
  creates two-dimensional histogram with double (64-bit) bin contents. Returns pointer to histogram object.
• PSHist::H2* hist2f(name, title, xaxis, yaxis)
  creates two-dimensional histogram with float (32-bit) bin contents. Returns pointer to histogram object.
• PSHist::Profile* prof1(name, title, xaxis, const std::string& option="")
  creates profile histogram, option string can be empty, "s", or "i", for meaning see reference. Returns pointer to histogram object.

User code should store the returned histogram pointers (as the module data members) and use is later in the code, there is no way currently to retrieve a pointer to the histogram created earlier.

Here is an example of the correct use of the histogramming package (from psana_examples.EBeamHist module):

h1. Histogramming Service

Psana includes a histogramming service which is wrapper for ROOT histogramming package. This service simplifies several tasks such as opening ROOT file, saving histograms to file, etc.

Center piece of the histogramming service is the histogram manager class. Histogram manager's responsibilities is to open ROOT file, create histograms, and to store histograms to the file. All these tasks are performed transparently to user, there is no need for additional configuration of this service. To create histograms one needs first to obtain a reference to a manager instance which is a part of the standard psana environment and is accessible through a method of the environment class. One then can call factory methods of the manager class to create new histograms which will be automatically saved to a ROOT file. The manager creates a single ROOT file to store all histograms created in a single job. Then name of the ROOT file is the same as the job name with ".root" extension added. The name of psana job is auto-generated from the name of the first input file, but it can also be set on the command line with {{\-j <job-name>}} option.

All factory methods of the histogram manager use special class to describe histogram axis (or axes for 2-dim histograms). The name of the class is {{PSHist::Axis}} (in the user module {{PSHist::}} prefix is optional) and it contains binning information for single histogram axis. It can be constructed in two different ways:
* {{Axis(int nbins, double amin, double amax)}}
\\
defines axis with fixed-width bins in the range from {{amin}} to {{amax}}.
* {{Axis(int nbins, const double\* edges)}}
\\
defines axis with variable-width bins, array contains the low edge of each bin plus high edge of the last bin. Total size of the edges array must be {{nbins+1}}.


Here is the list of the factory methods (see also [reference|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1HManager.html] for more information):
* {{PSHist::H1\* hist1i(const std::string& name, const std::string& title, const Axis& axis)}}
\\
creates [one-dimensional histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H1.html] with integer bin contents. Returns pointer to histogram object.
* {{PSHist::H1\* hist1d(name, title, axis)}}
\\
(argument types same as above) creates [one-dimensional histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H1.html] with double (64-bit) bin contents. Returns pointer to histogram object.
* {{PSHist::H1\* hist1f(name, title, axis)}}
\\
creates [one-dimensional histogram|http://https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H1.html] with float (32-bit) bin contents. Returns pointer to histogram object.
* {{PSHist::H2\* hist2i(name, title, xaxis, yaxis)}}
\\
creates [two-dimensional histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H2.html] with integer bin contents. Returns pointer to histogram object.
* {{PSHist::H2\* hist2d(name, title, xaxis, yaxis)}}
\\
creates [two-dimensional histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H2.html] with double (64-bit) bin contents. Returns pointer to histogram object.
* {{PSHist::H2\* hist2f(name, title, xaxis, yaxis)}}
\\
creates [two-dimensional histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1H2.html] with float (32-bit) bin contents. Returns pointer to histogram object.
* {{PSHist::Profile\* prof1(name, title, xaxis, const std::string& option="")}}
\\
creates [profile histogram|https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-doxy/html/classPSHist_1_1Profile.html], option string can be empty, "s", or "i", for meaning see reference. Returns pointer to histogram object.

User code should store the returned histogram pointers (as the module data members) and use is later in the code, there is no way currently to retrieve a pointer to the histogram created earlier.

Here is an example of the correct use of the histogramming package (from psana_examples.EBeamHist module):
{code}
// ==== EBeamHist.h ====
class EBeamHist: public Module {
public:
  .....
private:
  Source m_ebeamSrc;
  PSHist::H1* m_ebeamHisto;
  PSHist::H1* m_chargeHisto;
};

// ==== EBeamHist.cpp ====
EBeamHist::EBeamHist(const std::string& name)
  : Module(name)
  , m_ebeamHisto(0)
  , m_chargeHisto(0)
{
  m_ebeamSrc = configStr("eBeamSource", "BldInfo(EBeam)");
}

void EBeamHist::beginJob(Env& env)
{
  m_ebeamHisto = env.hmgr().hist1i("ebeamHisto", "ebeamL3Energy value", Axis(1000, 0, 50000));
  m_chargeHisto = env.hmgr().hist1i("echargeHisto", "ebeamCharge value", Axis(250, 0, 0.25));
}

void EBeamHist::event(Event& evt, Env& env)
{
  shared_ptr<Psana::Bld::BldDataEBeamV1> ebeam = evt.get(m_ebeamSrc);
  if (ebeam.get()) {
    m_ebeamHisto->fill(ebeam->ebeamL3Energy());
    m_chargeHisto->fill(ebeam->ebeamCharge());
  }
}
{code}

More

...

extensive

...

example

...

is

...

available

...

in

...

Psana

...

User

...

Examples

...

.

Writing User Modules

Here are few simple steps and guidelines which should help users to write their analysis modules.

• Everything is done in the context of the off-line analysis releases, your environment should be prepared and you should have test release setup based on one of the recent analysis releases. Consult Workbook which should help you going.
• You need your own package which may host several analysis modules. Package name must be unique. If the package has not be created yet run this command:

  Code Block
  newpkg MyPackage mkdir MyPackage/include MyPackage/src

...

• Generate skeleton module class from template:

  Code Block
  codegen -l psana-module MyPackage MyModule

...

• this

...

• will

...

• create

...

• two

...

• files:

...

• MyPackage/include/MyModule.h

...

• and

...

• MyPackage/src/MyModule.

...

• cpp
• Edit these two files,

...

• add

...

• necessary

...

• data

...

• members

...

• and

...

• implementation

...

• of

...

• the

...

• methods.

...

• For

...

• examples

...

• of

...

• accessing

...

• different

...

• data

...

• types

...

• see

...

• collection

...

• of

...

• modules

...

• in

...

• psana_examples

...

• package.

...

• Reference

...

• for

...

• all

...

• event

...

• and

...

• configuration

...

• data

...

• types

...

• is

...

• located

...

• at

...

• https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psddl_psana/

...

• Reference

...

• for

...

• other

...

• classes

...

• in

...

• psana

...

• framework:

...

• Psana

...

• Reference

...

• Manual
• Run scons to build the module library.
• Create psana config file if necessary.
• Run psana providing input data, configuration file, etc.
• It is also possible that somebody wrote a module which you can reuse for your analysis, check the module catalog: Psana Module Catalog

To add your own compiler or linker options to the build (such as to link to a third party library), see this section on customizing the scons build.

Running Psana

After writing and compiling the modules (or choosing standard modules) one can run psana application with these modules. Psana application is pre-built and does not need to be recompiled. To start application one needs to either provide a configuration file or corresponding command-line options. Some information (e.g. user module options) cannot be specified on the command line and always require configuration file. Here is the list of command-line options recognized by psana:

]
* Run {{scons}} to build the module library.
* Create psana config file if necessary.
* Run {{psana}} providing input data, configuration file, etc.
* It is also possible that somebody wrote a module which you can reuse for your analysis, check the module catalog: [Psana Module Catalog]

To add your own compiler or linker options to the build (such as to link to a third party library), see this [section on customizing the scons build|SConsTools#SconsCustomBuild].


h1. Running Psana

After writing and compiling the modules (or choosing standard modules) one can run psana application with these modules. Psana application is pre-built and does not need to be recompiled. To start application one needs to either provide a configuration file or corresponding command-line options. Some information (e.g. user module options) cannot be specified on the command line and always require configuration file. Here is the list of command-line options recognized by psana:
{code}
Usage: psana [options] [dataset ...]

  Available options:
    {-h|-?|--help    }         print help message
    {-v|--verbose    } (incr)  verbose output, multiple allowed (initial: 0)
    {-q|--quiet      } (incr)  quieter output, multiple allowed (initial: 2)
    {-b|--calib-dir  } path    calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "")
    {-c|--config     } path    configuration file, by default use psana.cfg if it exists (default: "")
    {-e|--experiment } string  experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "")
    {-j|--job-name   } string  job name, default is to generate from input file names (default: "")
    {-m|--module     } name    module name, more than one possible
    {-n|--num-events } number  maximum number of events to process, 0 means all (default: 0)
    {-s|--skip-events} number  number of events to skip (default: 0)
    {-o|--option     } string  configuration options, format: module.option[=value]

  Positional parameters:
    dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...)
{code}

If

...

both

...

options

...

-c

...

and

...

-m

...

are

...

missing

...

from

...

the

...

command

...

line

...

then

...

psana

...

reads

...

configuration

...

file

...

psana.cfg

...

from

...

current

...

directory.

...

Otherwise

...

if

...

-c

...

option

...

is

...

provided

...

with

...

the

...

file

...

name

...

psana

...

reads

...

corresponding

...

configuration

...

file.

...

Modules

...

loaded

...

by

...

psana

...

can

...

be

...

specified

...

in

...

configuration

...

and

...

on

...

command

...

line

...

with

...

-m

...

option.

...

If

...

-m

...

option

...

is

...

provided

...

then

...

its

...

value

...

overrides

...

module

...

list

...

specified

...

in

...

the

...

configuration

...

file.

...

One

...

can

...

provide

...

comma-separated

...

list

...

of

...

module

...

names

...

or

...

multiple

...

-m

...

options

...

on

...

the

...

command

...

line,

...

following

...

command

...

lines

...

are

...

all

...

equivalent:

}
% psana -m ModuleA,ModuleB,ModuleC ...
% psana -m ModuleA -m ModuleB -m ModuleC ...
% psana -m ModuleA,ModuleB -m ModuleC ...
{code}

Option {{\

Option -j

...

can

...

change

...

job

...

name

...

which

...

defines

...

then

...

names

...

of

...

the

...

output

...

histogram

...

file.

...

By

...

default

...

job

...

name

...

is

...

constructed

...

from

...

the

...

name

...

of

...

the

...

first

...

input

...

file.

...

Input

...

data

...

files

...

can

...

also

...

be

...

specified

...

in

...

the

...

configuration

...

file

...

or

...

on

...

command

...

line,

...

command-line

...

arguments

...

override

...

configuration

...

file

...

values.

...

Command-line

...

options

...

-v

...

and

...

-q

...

can

...

increase

...

or

...

decrease

...

verbosity

...

of

...

the

...

output

...

generated

...

by

...

messaging

...

service.

...

By

...

default

...

psana

...

outputs

...

messages

...

at

...

info

...

and

...

higher

...

levels.

...

With

...

one

...

-v

...

option

...

trace

...

messages

...

will

...

be

...

printed

...

also,

...

and

...

with

...

two

...

or

...

more

...

-v

...

options

...

debug

...

messages

...

will

...

be

...

printed

...

too.

...

With

...

-q

...

option

...

info

...

messages

...

will

...

not

...

be

...

printed,

...

only

...

warning

...

,

...

error

...

,

...

and

...

fatal

...

.

...

Here

...

are

...

few

...

examples

...

of

...

running

...

psana

...

applications:

}
% psana -m EventKeys /reg/d/psdm/...
% psana -m psana_examples.EBeamHist -j ebeam-hist-r1000 /reg/d/psdm/...
% psana -c psana_examples/data/DumpAll.cfg /reg/d/psdm/...
% psana                  # everything will be specified in psana.cfg file
{code}



h1. Psana Module Examples

A set of psana modules is available in current release as explained in [PCDS:Psana Module Catalog]. Part of them demonstrates how data can be accessed from user module code . Other modules can be used in data analysis or event filtering. Example of application for these modules are available in separate document:
* [PCDS:Psana Module Examples] \- for advanced modules for analysis and event filtering

We permanently work on algorithms implemented in the standard set of the psana modules. If you find that the algorithm which you need is missing in our collection you have two options:
* write it yourself using prescriptions from this 

Psana Module Examples

A set of psana modules is available in current release as explained in Psana Module Catalog. Part of them demonstrates how data can be accessed from user module code . Other modules can be used in data analysis or event filtering. Example of application for these modules are available in separate document:

• Psana Module Examples - for advanced modules for analysis and event filtering

We permanently work on algorithms implemented in the standard set of the psana modules. If you find that the algorithm which you need is missing in our collection you have two options:

• write it yourself using prescriptions from this document,