Content
Introduction
Offline analysis software for LCLS data historically pass a few stages;
- myana – the first C-based package,
- pyana – python-based framework,
- psana – C++-based framework.
Currently, features of both frameworks are combined together under the unified - psana – framework, which works both with C++ and python modules.
This new framework allows data processing using advantage of both C++ and python modules. However, there is no full backward compatibility between new psana framework and pyana modules. Original pyana modules would not work directly in psana, they need to be changed. In this note we discuss the difference between pyana and python-psana modules and present the references to documentation, which help to migrate from pyana to python-psana.
Data types and access methods
LCLS data types have different implementation in pyana and python-psana. Auto-generated from source code documentation is available in
- https://pswww.slac.stanford.edu/swdoc/releases/ana-current/pyana-ref/html/ for pyana and
- https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-ref/html/ for python-psana.
From this documentation it is clear that there is no full backward compatibility between these libraries; data types and access methods are different. Below we present a list of references with code examples for both frameworks.
Constructor arguments and configuration data
Pyana
Original pyana design used constructor arguments to pass configuration parameters to user modules. Typical user module defined its __init__
method with a number of arguments with many or all of them having default values:
class user_module(object): def __init__(self, source, threshold="0.25", max_count="1000"): self.source = source self.threshold = float(threshold) self.max_count = int(max_count)
The arguments that do not provide default values must be specified in the configuration file, arguments with the default values can be overriden by the values from configuration file. Suppose that configuration files contains this:
[user_package.user_module] source = SrcString threshold = 1.25
Then psana will instantiate user module with the parameters user_module(source="SrcString", threshold="1.25")
and max_count
argument will use default value from method declaration. Note that all arguments are passed as string values, no conversion to integers or floating point numbers is performed.
Starting with the release ana-0.9.6
pyana was modified to support different style of access to configuration information. In addition to receiving configuration parameters through constructor arguments one can also use few special methods of the module class to obtain the values of the parameters. These methods are:
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="6d2f8283-b5b1-4b62-9801-e2f9247afced"><ac:plain-text-body><![CDATA[ |
|
returns value of parameter as a boolean value, strings "yes", "true", "True", "on", "1" represent true value, strings "no", "false", "False", "off", "0" represent false value, any other string will raise exception. If parameter is not defined in a file then default value is returned without conversion, if default value was not given then exception is raised. |
]]></ac:plain-text-body></ac:structured-macro> |
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="3b024bf3-f899-42af-90ac-3b12dcb8e704"><ac:plain-text-body><![CDATA[ |
|
returns value of parameter as integer value. If parameter is not defined in a file then default value is returned without conversion, if default value was not given then exception is raised. If conversion from string to integer fails the standard exception is raised. |
]]></ac:plain-text-body></ac:structured-macro> |
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="92c576b7-7476-4b90-9e97-41f636ddae10"><ac:plain-text-body><![CDATA[ |
|
returns value of parameter as floating point value. If parameter is not defined in a file then default value is returned without conversion, if default value was not given then exception is raised. If conversion from string to floating point fails the standard exception is raised. |
]]></ac:plain-text-body></ac:structured-macro> |
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="767633b5-1878-4027-a7a7-5c6077c8a673"><ac:plain-text-body><![CDATA[ |
|
returns value of parameter as string. If parameter is not defined in a file then default value is returned without conversion, if default value was not given then exception is raised. |
]]></ac:plain-text-body></ac:structured-macro> |
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="c937ca88-be9f-45be-b275-82c671f38eef"><ac:plain-text-body><![CDATA[ |
|
returns value of parameter as "source address" string. If parameter is not defined in a file then default value is returned without conversion, if default value was not given then exception is raised. This method does the same as |
]]></ac:plain-text-body></ac:structured-macro> |
|
returns value of parameter as a list of boolean values. If parameter is not defined in a file then empty list is returned, otherwise every word in a parameter value is converted to boolean (according to same rules as defined for |
||
|
returns value of parameter as a list of integers. If parameter is not defined in a file then empty list is returned, otherwise every word in a parameter value is converted to integers and all numbers are returned in one list. Conversion errors will raise exception. |
||
|
returns value of parameter as a list of floating point numbers. If parameter is not defined in a file then empty list is returned, otherwise every word in a parameter value is converted to float and all numbers are returned in one list. Conversion errors will raise exception. |
||
|
returns value of parameter as a list of strings. If parameter is not defined in a file then empty list is returned, otherwise parameter value is split into words which are returned in one list. |
||
|
returns value of parameter as a list of "source address" strings. If parameter is not defined in a file then empty list is returned, otherwise parameter value is split into words which are returned in one list. |
These methods are "magic" in a sense that they are not defined by the module itself but instead are generated by the framework.
As the values of the configuration parameters can be obtained from the above methods defining arguments in the constructor is no longer necessary. pyana now supports modules which have empty argument list of the __init__
method (except for self
argument). The constructor of the example module above can be now defined as:
class user_module(object): def __init__(self): self.source = self.configStr('source') # no default - must be defined in a config file self.threshold = self.configFloat('threshold', 0.25) self.max_count = self.configInt('max_count', 1000)
This style should be preferred in pyana and it provides easier migration to psana.
Psana
The situation with the arguments and configuration parameters in psana is opposite to the pyana situation - Pyhton modules in psana are not supposed to define any arguments for the constructor and they should use methods defined above to access values of the configuration parameters. Like in pyana these methods are "magic" as they are defined by the framework itself and not by the module itself or its base classes. Unlike in pyana the methods self.configSrc()
and self.configListSrc()
return instances of psana.Source
class instead of string, this class is used as an argument to evt.get()
method in psana.
As a compatibility feature psana supports "old pyana style" of the constructor which receives configuration parameters via constructor arguments. This means that old pyana modules can work unchanged in this respect (but will still require changes for data access). This compatibility feature may be removed in the future as it may have unexpected implications, so it is much better to switch to the new psana style, especially that pyana now also supports this style.
Access to data in pyana and psana
There are very significant differences between pyana and psana in how the data are retrieved from the framework (via event and environment structues) and the format of the data classes. psana shares framework and data definition with C++ implementation and many things are closer to C++ interface than to the pyana (which is inspired by pdsdata). Here we discuss differences in retrieving the data.
Event data access in pyana
Pyana was designed based on pdsdata and myana and originally provided methods of data access which looked similar to those defined in myana. In particular event object provided a number of specialized methods for some (but not all) individual data types, e.g. evt.getAcqValue(address,channel,env)
for accessing Acqiris data, evt.getFeeGasDet()
for accessing FEE gas detector data. In addition to these specialized methods there is a generic method evt.get(typeId, address)
which can be used to retrieve any type of data. There are two "overloads" of this method:
evt.get(TypeId, address)
which takes integer number for the first argument and "source address" string for second argument. Integer number is an XTC TypeId value such asxtc.TypeId.Type.Id_Frame
. This method retrieves detector data of the type corresponding to the TypeId coming from a device that matches source address.evt.get(key[, default=None])
which takes a string for the first argument and optionally any object for second argument. This method retrieves "user data" object from event which was stored in the event by user module with by callingevt.put(object, key)
with the identical key.
Best way to access event data in pyana is to use generic evt.get()
method and not specialized methods, it would be much easier to migrate this code later to psana. Here is an example of this:
from pypdsdata import xtc ...... src = self.configSrc('source', '') data = evt.get(xtc.TypeId.Type.Id_FEEGasDetEnergy, src) if data: print " f_11_ENRC =", data.f_11_ENRC
Event data access in psana
Psana does not provide any specialized methods for retrieving individual data types, instead it provides generic evt.get()
method whose interface is much closer to that of C++ class, though it also provides few overloads for pyana compatibility. There are several "overloads" of this method depending on the number and types of parameters:
evt.get(type, src[, key:string])
wheretype
is python class corresponding to the data type (or list of classes), e.g.psana.Camera.FrameV1
.src
argument must be an instance ofpsana.Source
type which provides a match for device address,psana.Source
is returned fromself.configSrc(...)
method. Ifkey
argument is provided then it must be a string, missing key argument is identical to empty string.evt.get(type[, key:string])
- variation of the above method without source argument. This method is used to retrieve data which is not associated with any device but is a property of the event as a whole. Good example of this kid of data is thepsana.EventId
object.self.get(typeId:int, addr:string)
- this is pyana-compatibility method and should only be used during migration or in the code which is supposed to run in both pyana and psana. This method is equivalent to pyana methodevt.get(TypeId, address)
. Note that in psana there will be data types which do not have corresponding TypeId, this method cannot be used with those types (TypeId is a property of XTC data types, other data types do not have typeId).self.get(key:string)
- this is pyana-compatibility method and should only be used during migration or in the code which is supposed to run in both pyana and psana. This method is equivalent to pyana methodevt.get(key, None)
and is used to retrieve data stored withevt.put(object, key)
.
psana example corresponding to the above pyana example which retrieves gas detector data should look like:
# this is psana code, will not work in pyana import psana .... src = self.configSrc('source', '') data = evt.get(psana.Bld.BldDataFEEGasDetEnergy, src) if data: print " f_11_ENRC =", data.f_11_ENRC() # note method call
If one wants to write the code that should work in both pyana and psana then some modification of the pyana example should do:
from pypdsdata import xtc ...... src = self.configSrc('source', '') data = evt.get(xtc.TypeId.Type.Id_FEEGasDetEnergy, src) if data: if env.fwkName() == 'pyana': print " f_11_ENRC =", data.f_11_ENRC else: print " f_11_ENRC =", data.f_11_ENRC()
Note that there is a code specific to each framework and this is related to the differences in the data object interfaces.
Access to configuration objects in pyana
Just like event class the environment class in pyana provides a number of specific method for individual data types and a generic method which works for any data type. Specific methods should not be used frequently, one should prefer to use generic method. Generic method has this definition:
env.getConfig(typeId:int[, address=None])
- takes integer number for the first argument and optional string for second. Integer number is an XTC TypeId value such asxtc.TypeId.Type.Id_AcqConfig
(note that TypeId values for event data and configuration data are different). This method retrieves configuration data of the type corresponding to the TypeId coming from a device that matches source address.
An example:
from pypdsdata import xtc ...... src = self.configSrc('source', '') config = env.getConfig(xtc.TypeId.Type.Id_AcqConfig, src) if config: print "config.nbrChannels =", config.nbrChannels()
Access to configuration objects in psana
To access configuration data in psana there are two methods available: "true psana" and pyana-compatibility.
For code which runs in psana only it is better to use "true psana" method which consists in first getting access to special "config store" object and second retrieving data from that store object. To acees store object one should call env.configStore()
method without parameters which returns the instance of the EnvObjectStore class. This class defines get()
method which is a simplified version of evt.get()
method and has these overloads:
store.get(type, src)
wheretype
is python class corresponding to the configuration data type (or list of classes), e.g.psana.Acqiris.Config
.src
argument must be an instance ofpsana.Source
type which provides a match for device address.self.get(typeId:int[, address:string=""])
- this is pyana-compatibility method and should only be used in the code which is supposed to run in both pyana and psana. This method is equivalent to pyana methodenv.getConfig(TypeId, address)
.
For pyana compatibility environment also defines evt.getConfig(...)
method which is a shortcut for env.configStore().get(...)
, this method should only be used in a code which should run in both pyana and psana.
Here is a psana example for accessing the same Acqiris configuration data:
import psana ...... src = self.configSrc('source', '') config = env.configStore().get(psana.Acqiris.Config, src) if config: print "config.nbrChannels =", config.nbrChannels()
Previous pyana example is supposed to work in psana without change.
Differences in data classes
Tables below provide summary of differences between data objects inside pyana and psana.
For pyana the pieces of code imply that following import statement appears somewhere in the module file:
from pypdsdata import xtc
for psana this import should be replaced with
import psana
In pyana evt.get()
and env.getConfig()
accept string for source
argument while in psana the type of the argument must be psana.Source
. In both pyana and psana the source address can be set with self.configSrc()
as this method returns correct type in each framework:
self.m_src = self.configSrc('source', '')
As it is discussed above pyana code can be run inside psana in compatibility mode. In this case the code needs changes only if data classes provide different interfaces.
Acqiris
pyana |
psana |
---|---|
|
|
|
|
access to data also differs... |
|
Bld
pyana |
psana |
---|---|
|
|
|
|
|
|
|
|
CSPad
pyana |
psana |
---|---|
{{}} |
{{}} |
CSPad2x2
pyana |
psana |
---|---|
{{}} |
{{}} |
Epics
pyana |
psana |
---|---|
{{}} |
{{}} |
Evr
pyana |
psana |
---|---|
{{}} |
{{}} |
Pnccd
pyana |
psana |
---|---|
{{}} |
{{}} |
Princeton
pyana |
psana |
---|---|
{{}} |
{{}} |
Ipimb
pyana |
psana |
---|---|
{{}} |
{{}} |
Timepix
pyana |
psana |
---|---|
{{}} |
{{}} |
pyana |
psana |
---|---|
{{}} |
{{}} |
Interactive psana
Interactive psana is available beginning from release ana-0.9.0.
Example of how to start interactive python-psana:
% ipython from psana import * help(psana.CsPad)
References
- Auto-generated documentation for pyana modules: https://pswww.slac.stanford.edu/swdoc/releases/ana-current/pyana-ref/html/
- Auto-generated documentation for python modules in psana: https://pswww.slac.stanford.edu/swdoc/releases/ana-current/psana-ref/html/