Beam Test Pipeline Overview
The pipeline automatically retrieves all online data produced by LATTE which are currently stored in directories associated with runs numbers and brings them to the SLAC farm. After that, it populates an ORACLE database which provide queries to the data. The pipeline also creates reports, and launches data processing/reconstruction code to produce data files and high level analysis ntuples.
A pipeline diagram can be seen below 
Note that this diagram is not current, the recon task is more complex now. Rectangles represent TPs, the first one in a task is shaded. Ellipses represent files, while hexagons represent symbolic links.
The xml files used for upload are located in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
They are generated by running scripts when installing the code.
Need an explanation of how the tasks get launched by FastCopy
Some existing documentation can be found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current/doc.
install.txt and operation.txt may both be read directly. Running make in the doc directory will use IAndTPipeline.tex and several .dot files to make a .pdf.
Policy for updating tasks
Describes policy and steps for updating
Environmental Variables
Setting up the environment so that one can acess the ORACLE database
Source /u/gl/glast/pdb_config/dpf_config_prod.csh (or .sh if you use BASH) to run the pipeline text-mode management tools, which are installed in $PDB_HOME (which is set by the config script).
Pipeline Tasks and Associated Scripts
Each pipeline tasks consists of several shell, python and perl scripts.
Most steps within a task (Task Processes (TPs)) consist of a wrapper, which interfaces with the pipeline infrastructure (GINO) and a script which does the work, usually by running some external app.
The code for the bt pipeline is found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
The cvs repository for the code is here. It contains a branch with the following features:
- based on SVAC pipeline
- minimal changes necessary to make it work
The list of pipeline taks is provided below (in the alphabetical order) with the information how to run them.
Task Name: configReport
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
start ConfigTables task |
configTablesWrapper.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
Task Name: digiReport
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
genDigiTestReport.pl |
|
|
|
start ConfigTables task |
genDigiTestReportWrapper.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
Task Name: digitization
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
determine whether to reconstruct run |
decideRecon.pl |
|
|
|
start digitization task |
genDTRLaunchWrapper.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
|
ldfToDigi.pl |
|
|
pipeline side |
enter run in eLog |
ldfToDigiWrapper.pl |
|
|
external side |
enter run in eLog |
recLaunchWrapper.pl |
|
|
pipeline side |
|
reprocess-v2r0.csh |
|
|
|
|
eprocess.csh |
|
|
|
|
reprocessEM2.csh |
|
|
|
|
retDefToDigiWrapper.pl |
|
|
|
|
setEvents.pl |
|
|
|
|
setEventsWraspper.pl |
|
|
|
Task Name: eLogupdate
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
|
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
|
retDefTDLaunchWrapper.pl |
|
|
not used by beamtest |
Task Name: offLineTest
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
setup.csh |
|
|
|
start ConfigTables task |
test.pl |
|
|
|
Task Name: online
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
BeamTestLaunch.pl |
|
|
not used |
start ConfigTables task |
LicosLaunch.pl |
|
|
|
determine whether to digitize run |
SVACLaunch.pl |
|
|
|
make XML file defining task |
SVACWrapper.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
getAlgFile.pl |
|
|
pipeline side |
Task Name: reconReport
Description: lo
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
genReconTestReport.pl |
|
|
|
start ConfigTables task |
genReconTestReportWrapper.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
Task Name: svacTuple
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
|
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
Task Name: lib
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
|
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
|
retDefTDLaunchWrapper.pl |
|
|
not used by beamtest |
Task Name: recon
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
|
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
|
retDefTDLaunchWrapper.pl |
|
|
not used by beamtest |
Task Name: setup
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
|
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
|
retDefTDLaunchWrapper.pl |
|
|
not used by beamtest |