You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 29 Next »


Beam Test Pipeline Overview

Currently being updated
The pipeline automatically retrieves all online data produced by LATTE which are currently stored in directories associated with runs numbers and brings them to the SLAC farm. After that, it populates an ORACLE database which provide queries to the data. The pipeline also creates reports, and launches data processing/reconstruction code to produce data files and high level analysis ntuples.
A pipeline diagram can be seen below

Note that this diagram is not current, the recon task is more complex now.  Rectangles represent TPs, the first one in a task is shaded.  Ellipses represent files, while hexagons represent symbolic links.

The xml files used for upload are located in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
They are generated by running scripts when installing the code. 

Need an explanation of how the tasks get launched by FastCopy

Some existing documentation can be found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current/doc.
install.txt and operation.txt may both be read directly.  Running make in the doc directory will use IAndTPipeline.tex and several .dot files to make a .pdf.

Policy for updating tasks

Describes policy and steps for updating

Environmental Variables

Setting up the environment so that one can acess the ORACLE database

Source  /u/gl/glast/pdb_config/dpf_config_prod.csh (or .sh if you use BASH) to run the pipeline text-mode management tools, which are installed in $PDB_HOME (which is set by the config script).

Pipeline Tasks and Associated Scripts

Each pipeline task consists of several c-shell, python and perl scripts.

Pipeline tasks are structured as a linear chain of steps.  Each step must succeed before going on to the next one.  Each step must be attempted, until one fails or the last one succeeds.  That is to say, there is no flow control, and no parallelism.  In order to deal with this, and to reduce the chance that a step would not be reached due to the failure of a previous step on which it did not logically depend, the work we do in the pipeline is split into 7 tasks, all but one of which (updateELogDB) are launched by other tasks.  This launching is done without GINO's (the pipeline infrastructure) knowledge.

Most steps within a task (Task Processes (TPs)) consist of a wrapper, which interfaces with GINO and a script which does the work, usually by running some external app.

The code for the bt pipeline is found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current

The cvs repository for the code is here. It contains a branch with the following features:

  • based on SVAC pipeline
  • minimal changes necessary to make it work

The list of pipeline taks is provided below (in the alphabetical order) with the information how to run them.

Task Name: configReport

Description:

 Purpose

 Associated Scripts

Input

 Output

Comments

make config report

 configTablesWrapper.pl

 

 

 pipeline side

make XML file defining task

 genXml.pl

 

 

 currently makes 2 XML files, LICOS version should be removed

Task Name: digiReport

Description:

 Purpose

 Associated Scripts

Input

 Output

Comments

 make digi report

 genDigiTestReport.pl

 

 

 external side

make digi report

 genDigiTestReportWrapper.pl

 

 

 pipeline side

make XML file defining task

 genXml.pl

 

 

 

Task Name: digitization

Description:

 Purpose

 Associated Scripts

Input

 Output

Comments

determine whether to reconstruct run

 decideRecon.pl

 

 

 

start digitization task

 genDTRLaunchWrapper.pl

 

 

 

make XML file defining task

 genXml.pl

 

 

 currently makes 2 XML files, LICOS version should be removed


 ldfToDigi.pl

 

 

 external side

enter run in eLog

 ldfToDigiWrapper.pl

 

 

 pipeline side

enter run in eLog

 recLaunchWrapper.pl

 

 

 pipeline side

 

 reprocess-v2r0.csh

 

 

  delete

 

  reprocess.csh

 

 

 

 

 reprocessEM2.csh

 

 

 delete

 

 retDefToDigiWrapper.pl

 

 

LICOS stuff, not used by beamtest

 

 setEvents.pl

 

 

  LICOS stuff, not used by beamtest

 

 setEventsWraspper.pl

 

 

LICOS stuff, not used by beamtest

Task Name: eLogupdate

Description: loads the database

 Purpose

 Associated Scripts

Input

 Output

Comments

 

 archiveWrapper.pl

 

 

 not used

start ConfigTables task

 ConfTLaunchWrapper.pl

 

 

 pipeline side

determine whether to digitize run

 decideDigi.pl

 

 

 

make XML file defining task

 genXml.pl

 

 

 currently makes 2 XML files, LICOS version should be removed

start digitization task

 ldfTDLaunchWrapper.pl

 

 

 pipeline side

enter run in eLog

 populateElogDb.pl

 

 

 external side

enter run in eLog

 populateElogDbWrapper.pl

 

 

 pipeline side

 

 retDefTDLaunchWrapper.pl

 

 

 not used by beamtest

Task Name: lib

Description: 

 Purpose

 Associated Scripts

Input

 Output

Comments

used by cleanupRecon.csh, gets rid of junk on one host

 _cleanup0ne.csh

 

 

  utility, not used in pipeline

clean up junk left on  local disks of batch hosts by failed chunks

  cleanupRecon.csh

 

 

 utility, not used in pipeline

   shared by all TPs that copy files

  copyWrapper.pl

 

 

 pipeline side

 reset eLog links to reports & ROOT files so there isn't a confusing mixed set of versions while reprocessing

  deleteLinks.csh

 

 

  utility, not used in pipeline

  shared by all TPs that delete files

  deleteWrapper.pl

 

 

 pipeline side

  merge tuple chunks into single file

  haddMerge.py

 

 

 external side

  merge tuple chunks into single file

  haddWrapper.pl

 

 

 pipeline side

 

  makeLinks.pl

 

 


 look things up in eLog

  queryElogReportTable.pl

 

 

 

 

  TaskLaunch.pl

 

 

 

 

  tryAFewTimes.csh

 

 

 

 change things in eLog

  updateElogReportTable.pl

 

 

 

 

  updateUrl.py

 

 

 

 

  urlWrapper.pl

 

 

 pipeline side

 

Task Name: offLineTest

Description: 

 Purpose

 Associated Scripts

Input

 Output

Comments

 

 setup.csh

 

 

 

start ConfigTables task

 test.pl

 

 

 

Task Name: online

Description: loads the database

 Purpose

 Associated Scripts

Input

 Output

Comments

launch a run

 BeamTestLaunch.pl

 

 

 


 LicosLaunch.pl

 

 

 not used

 

 SVACLaunch.pl

 

 

 not used

 

 SVACWrapper.pl

 

 

 not used


 getAlgFile.pl

 

 

  not used

 

Task Name: recon

Description: 

The recon task is relatively complex.  It splits a run into chunks, submits each one for reconstruction as a batch job (without GINO's knowledge), then merges the reconstructed chunks.

The first step, setupRecon, has almost all of the intelligence.  It decides how many chunks to use and which events will go in which chunk.  It writes a number of files which direct the action of later steps, including jobOptions files for reconstruction of the chunks.

doRecon controls the reconstruction of the chunks.  It reads a list of jobs to run, and for each one, spawns a thread that submits a batch job, waits for it to complete, and returns its status.  It then writes a file listing which, if any, of the chunks failed, and exits with an unsuccessful return code if there were failed chunks.  When the TP first starts, it checks for this list of failed chunks, and if it is present and nonempty, it uses it instead of the original list of all chunks writted by setupRecon.  This way, if some chunks fail, the TP can be rolled back and it will only need to redo the failed chunks.  The failed chunk list is not registered as a pipeline dataset, since it violates logical constraints within GINO for a TP to modify its input this way.

 In order to avoid problems with unreliable NFS service, the chunk jobs copy the input digi file to a local disk on the batch host (if it has one, I think they all do now) and writes the output files there as well.  It then moves the output files to a staging directory on AFS, and deletes the local copy of the input file.  When chunks fail, these files are left behind and eventually fill up the local disk.  Therefore, there is a script to seek out and delete these orphaned files.  It must run as user glastdpf.  I usually log into a noric as glastdpf and run it by hand every once in a while, but a better solution would probably be to wrap it in a task and run that task every night from my crontab.

Merging the chunks of the recon file uses 4 TPs.  The first (mergeRecon) performs the actual merge, from chunk files on the staging disk to a recon file on the staging disk.  The second deletes the chunk files from the staging disk.  The third copies the merged recon file from the staging disk to its final destination on NFS.  The fourth deletes the berged file from the staging area.  This may seem unreasonably complicated, but it reduces the amount 

 Purpose

 Associated Scripts

Input

 Output

Comments

 finish up a run

  cleanup.py

 

 

  external side

finish up a run

  cleanupWrapper.pl

 

 

 pipeline side

control job for reconstruction of chunks

  doRecon.pl

 

 

 external side

control job for reconstruction of chunks

  doReconWrapper.pl

 

 

  pipeline side

 

  genRTRLaunchWrapper.pl

 

 

  pipeline side

make XML config file for task

  genXml.pl

 

 

 

  merge chunks of recon file

  mergeRecon.py

 

 

external side

 merge chunks of recon file

  mergeReconWrapper.pl

 

 

  pipeline side

 

  recon.py

 

 

 obsolete

 reconstruct one chunk

  recon0ne.csh

 

 

 

 

  reconWrapper.pl

 

 

 obsolete

 

  reprocess-licos.csh

 

 

 delete

 

  reprocess-v3r1p5.csh

 

 

delete

 

  reprocess-version.csh

 

 

 delete

 launch recon report

  RunRALaunchWrapper.pl

 

 

 pipeline side

 prepare chunk jobs

  setupRecon.py

 

 

 external side

 prepare chunk jobs

  setupReconWrapper.pl

 

 

 pipeline side

Task Name: reconReport

Description: lo

 Purpose

 Associated Scripts

Input

 Output

Comments

 make recon report

 genReconTestReport.pl

 

 

 external side

make recon report

 genReconTestReportWrapper.pl

 

 

 pipeline side

make XML file defining task

 genXml.pl

 

 

 

 

Task Name: setup

Description:

 Purpose

 Associated Scripts

Input

 Output

Comments

setup for database access

  dbSetup8.cshrc

 

 

 Oracle 8, used by Python scripts

  setup for database access

  dbSetup10.cshrc

 

 

Oracle 10, used by Perl scripts

master setup file

  svacPlSetup.cshrc

 

 

  used by everything

 

Task Name: svacTuple

Description: loads the database

 Purpose

 Associated Scripts

Input

 Output

Comments

make beamtest tuple

 RunRootAnalyzer.pl

 

 

 external side

make beamtest tuple

 RunRootAnalyzerWrapper.pl

 

 

 pipeline side

make XML file defining task

 genXml.pl

 

 

 

 

 reprocess-licos-v3r5p5.pl

 

 

  delete


 reprocess-v3r4p6.csh

 

 

  delete


 reprocess.csh

 

 

  delete

 

 

  • No labels