Beam Test Pipeline

Beam Test Pipeline Overview

Currently being updated
The pipeline automatically retrieves all online data produced by LATTE which are currently stored in directories associated with runs numbers and brings them to the SLAC farm. After that, it populates an ORACLE database which provide queries to the data. The pipeline also creates reports, and launches data processing/reconstruction code to produce data files and high level analysis ntuples.
A pipeline diagram can be seen below

Note that this diagram is not current, the recon task is more complex now.  Rectangles represent TPs, the first one in a task is shaded.  Ellipses represent files, while hexagons represent symbolic links.

The xml files used for upload are located in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
They are generated by running scripts when installing the code. 

Need an explanation of how the tasks get launched by FastCopy

Some existing documentation can be found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current/doc.
install.txt and operation.txt may both be read directly.  Running make in the doc directory will use IAndTPipeline.tex and several .dot files to make a .pdf.

Policy for updating tasks

Describes policy and steps for updating

Environmental Variables

Setting up the environment so that one can acess the ORACLE database

Source  /u/gl/glast/pdb_config/dpf_config_prod.csh (or .sh if you use BASH) to run the pipeline text-mode management tools, which are installed in $PDB_HOME (which is set by the config script).

Pipeline Tasks and Associated Scripts

Each pipeline task consists of several c-shell, python and perl scripts.

Pipeline tasks are structured as a linear chain of steps.  Each step must succeed before going on to the next one.  Each step must be attempted, until one fails or the last one succeeds.  That is to say, there is no flow control, and no parallelism.  In order to deal with this, and to reduce the chance that a step would not be reached due to the failure of a previous step on which it did not logically depend, the work we do in the pipeline is split into 7 tasks, all but one of which (updateELogDB) are launched by other tasks.  This launching is done without GINO's (the pipeline infrastructure) knowledge.

Most steps within a task (Task Processes (TPs)) consist of a wrapper, which interfaces with GINO and a script which does the work, usually by running some external app.

The code for the bt pipeline is found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current

The cvs repository for the code is here. It contains a branch with the following features:

• based on SVAC pipeline

• minimal changes necessary to make it work

The list of pipeline taks is provided below (in the alphabetical order) with the information how to run them.

Task Name: configReport

Description:

Task Name: digiReport

Description:

Task Name: digitization

Description:

Task Name: eLogupdate

Description: loads the database

Task Name: lib

Description: 

Task Name: offLineTest

Description: 

Task Name: online

Description: loads the database

Task Name: recon

Description: 

The recon task is relatively complex.  It splits a run into chunks, submits each one for reconstruction as a batch job (without GINO's knowledge), then merges the reconstructed chunks.

The first step, setupRecon, has almost all of the intelligence.  It decides how many chunks to use and which events will go in which chunk.  It writes a number of files which direct the action of later steps, including jobOptions files for reconstruction of the chunks.

doRecon controls the reconstruction of the chunks.  It reads a list of jobs to run, and for each one, spawns a thread that submits a batch job, waits for it to complete, and returns its status.  It then writes a file listing which, if any, of the chunks failed, and exits with an unsuccessful return code if there were failed chunks.  When the TP first starts, it checks for this list of failed chunks, and if it is present and nonempty, it uses it instead of the original list of all chunks writted by setupRecon.  This way, if some chunks fail, the TP can be rolled back and it will only need to redo the failed chunks.  The failed chunk list is not registered as a pipeline dataset, since it violates logical constraints within GINO for a TP to modify its input this way.

 In order to avoid problems with unreliable NFS service, the chunk jobs copy the input digi file to a local disk on the batch host (if it has one, I think they all do now) and writes the output files there as well.  It then moves the output files to a staging directory on AFS, and deletes the local copy of the input file.  When chunks fail, these files are left behind and eventually fill up the local disk.  Therefore, there is a script to seek out and delete these orphaned files.  It must run as user glastdpf.  I usually log into a noric as glastdpf and run it by hand every once in a while, but a better solution would probably be to wrap it in a task and run that task every night from my crontab.

Merging the chunks of the recon file uses 4 TPs.  The first (mergeRecon) performs the actual merge, from chunk files on the staging disk to a recon file on the staging disk.  The second deletes the chunk files from the staging disk.  The third copies the merged recon file from the staging disk to its final destination on NFS.  The fourth deletes the berged file from the staging area.  This may seem unreasonably complicated, but it reduces the amount 

Task Name: reconReport

Description: lo

Task Name: setup

Description:

Task Name: svacTuple

Description: loads the database