The skimmer is only usable on linux. For what concerns the external tools, skimmer v6r0 depends on :
The basic task of the skimmer is to take Glast ROOT files, containing ROOT trees, and produce similar output files with a subset of branches and events. The search for ROOT data files to be skimmed is called here mining. The eventual non-copy of some branches is called pruning. The copy of only a subset of events is called filtering.
A skimmer execution is organized as a sequence of steps. All but the last one are preparation steps. Each preparation step is expected to prepare a specific parameter file. If the parameter file already exists, either from a previous execution or because it has been written from scratch by the user, then the corresponding preparations step is not applied (this default behavior can be changed). Currently, the steps are :
The Skimmer is also known to GLAST people as the Data Server Back End. It has a command-like interface which can be used directly, or it can also be used through a web interface, also known as the Data Server Front End, or Skimmer Web Application. Here, one will only find the documentation of the command-like interface, but this can also help to understand the corresponding web front-end.
The behavior of the skimmer is all tuned by some predefined shell variables. For a complete list of those variables, one can type "skimmer help", but the explanations will hardly make sense if you have not read this guide before.
As one can see in the steps given above, before the skimmer can proceed, it is collecting much information about the files to be skimmed, what they contain and what to extract. This is all tuned by some shell variables, and the information is mostly stored in few parameter files. The use of those text files bring several benefits :
All the parameter files can contain any number of empty lines and comments starting with "#". The lines starting with "#!" are called special comments. The first special comment in any parameter file should be about the global file format release, currently "CEL TXT 0.1". The second special comment should of the form "SECTION <name>", where <name> depends on the kind of information in the rest of the file. For example, if the file contain the list of input files, <name> will be "Files". Several examples will be given below.
One will find below the description of the parameter files and shell variables which are tuning a skimmer job.
Worth to note, for each of the official skimming step given previously, there is few systematic variables :
SK_SKIP_<STEP>
: do not execute the step, whatever the values of other parameters and existence of parameter files.SK_FORCE_<STEP>
: do execute the step, whatever the values of other parameters and existence of parameter files.SK_DEBUG_<STEP>
: will display additionnal information about that specific step.Obviously enough, for a given <STEP>
, SK_FORCE_<STEP>
and SK_SKIP_<STEP>
should never be true
at the same time. Let's know see the details of each step.
The list of input data files can be obtained by the skimmer from different sources :
SK_INPUT_CEL
.SK_INPUT_FILE_LIST
.SK_INPUT_TASK
.In the case of Pipeline I products, SK_INPUT_TASK
is enough and should be any of the tasks recognized by the Pipeline I Oracle Database. On top of that, one can select a subset of the task runs through the shell variables SK_RUN_MIN
and SK_RUN_MAX
. If SK_RUN_MAX
is set to 0
, all the runs will be taken into consideration.
In the case of a textual file made by the user, it must conform to the usual rules for the skimmer parameter files : the header is made of a first special comment which recall the global file format release (CEL TXT 0.1), and a special comment which declare that what follows is a list of files (SECTION Files). Then, each line of the file is the full path of an input ROOT file, eventually prefixed by the data types of the trees within the file. For example :
#
#! CEL TXT 0.1
#
#! SECTION Files
(recon)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000001/BeamTest-0100_000001_recon_RECON.root
(recon)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000002/BeamTest-0100_000002_recon_RECON.root
(mc)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000001/BeamTest-0100_000001_mc_MC.root
(mc)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000002/BeamTest-0100_000002_mc_MC.root
(merit:pointing:jobinfo)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000001/BeamTest-0100_000001_merit_merit.root
(merit:pointing:jobinfo)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000002/BeamTest-0100_000002_merit_merit.root
(digi)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000001/BeamTest-0100_000001_digi_DIGI.root
(digi)/nfs/farm/g/glast/u35/MC-tasks/BeamTest-0100/output/000002/BeamTest-0100_000002_digi_DIGI.root
Whatever the source for the list of input data files, one can obtain a copy of this list when giving a value to SK_OUTPUT_FILE_LIST
. This list is restrained to the files whose at least one entry has been kept after skimming. The ouput format is the same as the input format above. After a skimming, one can copy the output file defined by SK_OUTPUT_FILE_LIST
, edit it freely and reuse the copy later as a SK_INPUT_FILE_LIST
. It is not recommended to use a single file in such a scenario, because the file defined by SK_OUTPUT_FILE_LIST
is always overwritten, and you could loose your modifications.
Here are the default values of the shell variables for input data files mining (i.e. the values which are used if the variables are undefined or empty strings) :
SK_INPUT_CEL = "" SK_INPUT_FILE_LIST = "" SK_INPUT_TASK = "" SK_RUN_MIN = 0 SK_RUN_MAX = 0 SK_OUTPUT_FILE_LIST="" SK_DEBUG_FILE_LIST="false"
When managing data such as recon, mc and/or digis, the skimmer needs to load the corresponding C++ libraries, with the correct release, and it will take them in a dedicated file, whose name is defined by variable SK_LIBRARY_LIST_FILE
. In this file, each line is the full path of a shared library, eventually prefixed by its data type. Example of such a file :
#
#! CEL TXT 0.1
#
#! SECTION Libraries
/nfs/farm/g/glast/u09/builds/rh9_gcc32/BeamtestRelease/BeamtestRelease-v3r0907p0/lib/libcommonRootData.so
(mc)/nfs/farm/g/glast/u09/builds/rh9_gcc32/BeamtestRelease/BeamtestRelease-v3r0907p0/lib/libmcRootData.so
(digi)/nfs/farm/g/glast/u09/builds/rh9_gcc32/BeamtestRelease/BeamtestRelease-v3r0907p0/lib/libdigiRootData.so
(recon)/nfs/farm/g/glast/u09/builds/rh9_gcc32/BeamtestRelease/BeamtestRelease-v3r0907p0/lib/libreconRootData.so
This listcorresponds to a single code release, which is the one relevant for the current SK_TASK
. If a user is writing himself the file defined by SK_FILE_LIST_FILE
, he must be careful that all the listed ROOT files have been generated with the same code release.
If the file defined by SK_LIBRARY_LIST_FILE
does not already exists, the skimmer will try to guess the release and generate the lacking file. This guess is based on the eventual instance of FileHeader which is staying in the first file of each data kind. The latter can be modified by several tricky variables, but this is outside the scope of this guide (see the FAQ).
Here are the default values of the shell variables for this section :
SK_LIBRARY_LIST_FILE="${PWD}/${SK_TASK}_LibraryList.txt" SK_SKIP_GET_LIBRARY_LIST="false" SK_FORCE_GET_LIBRARY_LIST="false" SK_DEBUG_GET_LIBRARY_LIST="false"
The actual skimming will only select the events given in a parameter file, whose name is defined by the shell variable SK_EVENT_LIST_FILE
. Perhaps this file already exists, generated from a previous skimmer execution (and you eventually modified it), or because you wrote it from scratch. If not, the skimmer will generate one, based on the values of SK_TCUT_DATA_TYPE
and SK_TCUT
. The syntax of SK_TCUT
should be the ROOT one. Currently, the only valid value for SK_TCUT_DATA_TYPE
is merit
.
The expected content of the file defined by SK_EVENT_LIST_FILE
is a simple list of pairs, one by line, where each pair is made of a run id followed by an event id. The "events" section header can be followed by two special comments which recall the number of events before and after the cut (this information is currently not used by the skimmer). For example :
#
#! CEL TXT 0.1
#
#! SECTION Events
#! 2000 entries in original dataset.
#! 7 events after cut:
1 8
1 183
1 344
1 553
2 117
2 517
2 980
The possibility to define a TCut
which is going through several data types is not possible today, but it will be studied when we have several possible value for SK_TCUT_DATA_TYPE
. Here are the default values of the shell variables for this section :
SK_TCUT_DATA_TYPE="merit" SK_TCUT="" SK_EVENT_LIST_FILE="${PWD}/${SK_TASK}_EventList.txt" SK_SKIP_GET_EVENT_LIST="false" SK_FORCE_GET_EVENT_LIST="false" SK_DEBUG_GET_EVENT_LIST="false"
In the case you do want to keep all the entries, which is not the default expectation, you need to act on several variables : the value of SK_TCUT
should be ""
, the file defined by SK_EVENT_LIST_FILE
must not exist, and the value of SK_SKIP_GET_EVENT_LIST
should be "false"
. This way, the skimmer has no information about how to select the events, and will merge them all.
The skimmer can also take into account a list of the branches to be activated or desactivated. This list is given through a file, whose full file (including path) is given by variable SK_BRANCH_LIST_FILE
. If this file does not exists, skimmer will generate it, and activate all the branches.
As usual, you can edit this generated file, or write one from scratch. Each line should contains a data type prefix, the name of the tree, a {+} or a -
(so to activate or desactivate respectively), and the specification of one or several branches (with the ROOT syntax). The lines are applied one after the other : you can desactivate all the branches of a given type with *
, then activate the only ones of interest. For example :
#
#! CEL TXT 0.1
#
#! SECTION Branches
(recon)Recon+ReconEvent
(recon)Recon+m_eventId
(recon)Recon+m_runId
(recon)Recon+m_acd
(recon)Recon-m_cal
(merit)MeritTuple-*
(merit)MeritTuple+PtT*
(merit)MeritTuple+Cal*
Here are the default values of the shell variables for this section :
SK_BRANCH_LIST_FILE="${PWD}/${SK_TASK}_BranchList.txt" SK_SKIP_GET_BRANCH_LIST=false SK_FORCE_GET_BRANCH_LIST=false SK_DEBUG_GET_BRANCH_LIST=false
We are now to the point where to say which types of data we want to skim. This is said by shell variable SK_DATA_TYPES
, which should be a ":" separated list of data types. The current recognized types can be found in the guide /Skimmer at SLAC/. If SK_DATA_TYPES
is empty, a default value of "merit:mc:digi:recon" will be used.
The skimmed files will be stored in the directory defined by shell variable SK_OUT_DIR
, in files called SK_OUT_FILE_BODY_<datatype>.root
. Yet, if they turned to be very big files, ROOT could automatically close the first file and open new ones, appending a rank number to the file name. The maximum size of each ouput ROOT file can be changed with shell variable SK_MAX_FILE_SIZE
. If the value of 0 is given to this variable (this is the default), ROOT will use its own default value. Also, if the value is 0 and the job is merging all the events, the ROOT fast merging method will be used.
Here are the default values of the shell variables for this section :
SK_DATA_TYPES="merit:mc:digi:recon" SK_OUT_DIR=${PWD} SK_OUT_FILE_BODY=${SK_TASK} SK_MAX_FILE_SIZE=0 SK_SKIP_SKIM=false SK_DEBUG_SKIM=false
If one has already executed the skimmer, and the generated parameter files are still available, most of the skimmer shell variables will not affect the new execution, because the skimmer will directly reuse the parameter files. Only the few variables above, corresponding to the actual skimming step, will be taken into account.