|
|---|
Beam Test Pipeline Overview
Currently being updated
The pipeline automatically retrieves all online data produced by LATTE which are currently stored in directories associated with runs numbers and brings them to the SLAC farm. After that, it populates an ORACLE database which provide queries to the data. The pipeline also creates reports, and launches data processing/reconstruction code to produce data files and high level analysis ntuples.
A pipeline diagram can be seen below 
Note that this diagram is not current, the recon task is more complex now. Rectangles represent TPs, the first one in a task is shaded. Ellipses represent files, while hexagons represent symbolic links.
The xml files used for upload are located in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
They are generated by running scripts when installing the code.
Need an explanation of how the tasks get launched by FastCopy
Some existing documentation can be found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current/doc.
install.txt and operation.txt may both be read directly. Running make in the doc directory will use IAndTPipeline.tex and several .dot files to make a .pdf.
Policy for updating tasks
Describes policy and steps for updating
Environmental Variables
Setting up the environment so that one can acess the ORACLE database
Source /u/gl/glast/pdb_config/dpf_config_prod.csh (or .sh if you use BASH) to run the pipeline text-mode management tools, which are installed in $PDB_HOME (which is set by the config script).
Pipeline Tasks and Associated Scripts
Each pipeline task consists of several c-shell, python and perl scripts.
Pipeline tasks are structured as a linear chain of steps. Each step must succeed before going on to the next one. Each step must be attempted, until one fails or the last one succeeds. That is to say, there is no flow control, and no parallelism. In order to deal with this, and to reduce the chance that a step would not be reached due to the failure of a previous step on which it did not logically depend, the work we do in the pipeline is split into 7 tasks, all but one of which (updateELogDB) are launched by other tasks. This launching is done without GINO's (the pipeline infrastructure) knowledge.
Most steps within a task (Task Processes (TPs)) consist of a wrapper, which interfaces with GINO and a script which does the work, usually by running some external app.
The code for the bt pipeline is found in /afs/slac/g/glast/ground/PipelineConfig/BeamTest-tasks/beamtestPipeline/current
The cvs repository for the code is here. It contains a branch with the following features:
- based on SVAC pipeline
- minimal changes necessary to make it work
The list of pipeline taks is provided below (in the alphabetical order) with the information how to run them.
Task Name: configReport
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
make config report |
configTablesWrapper.pl |
|
|
pipeline side |
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
Task Name: digiReport
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
make digi report |
genDigiTestReport.pl |
|
|
external side |
make digi report |
genDigiTestReportWrapper.pl |
|
|
pipeline side |
make XML file defining task |
genXml.pl |
|
|
|
Task Name: digitization
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
determine whether to reconstruct run |
decideRecon.pl |
|
|
|
start digitization task |
genDTRLaunchWrapper.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
|
ldfToDigi.pl |
|
|
external side |
enter run in eLog |
ldfToDigiWrapper.pl |
|
|
pipeline side |
enter run in eLog |
recLaunchWrapper.pl |
|
|
pipeline side |
|
reprocess-v2r0.csh |
|
|
delete |
|
reprocess.csh |
|
|
|
|
reprocessEM2.csh |
|
|
delete |
|
retDefToDigiWrapper.pl |
|
|
LICOS stuff, not used by beamtest |
|
setEvents.pl |
|
|
LICOS stuff, not used by beamtest |
|
setEventsWraspper.pl |
|
|
LICOS stuff, not used by beamtest |
Task Name: eLogupdate
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
archiveWrapper.pl |
|
|
not used |
start ConfigTables task |
ConfTLaunchWrapper.pl |
|
|
pipeline side |
determine whether to digitize run |
decideDigi.pl |
|
|
|
make XML file defining task |
genXml.pl |
|
|
currently makes 2 XML files, LICOS version should be removed |
start digitization task |
ldfTDLaunchWrapper.pl |
|
|
pipeline side |
enter run in eLog |
populateElogDb.pl |
|
|
external side |
enter run in eLog |
populateElogDbWrapper.pl |
|
|
pipeline side |
|
retDefTDLaunchWrapper.pl |
|
|
not used by beamtest |
Task Name: lib
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
used by cleanupRecon.csh, gets rid of junk on one host |
_cleanup0ne.csh |
|
|
utility, not used in pipeline |
clean up junk left on local disks of batch hosts by failed chunks |
cleanupRecon.csh |
|
|
utility, not used in pipeline |
shared by all TPs that copy files |
copyWrapper.pl |
|
|
pipeline side |
reset eLog links to reports & ROOT files so there isn't a confusing mixed set of versions while reprocessing |
deleteLinks.csh |
|
|
utility, not used in pipeline |
shared by all TPs that delete files |
deleteWrapper.pl |
|
|
pipeline side |
merge tuple chunks into single file |
haddMerge.py |
|
|
external side |
merge tuple chunks into single file |
haddWrapper.pl |
|
|
pipeline side |
|
makeLinks.pl |
|
|
|
look things up in eLog |
queryElogReportTable.pl |
|
|
|
|
TaskLaunch.pl |
|
|
|
|
tryAFewTimes.csh |
|
|
|
change things in eLog |
updateElogReportTable.pl |
|
|
|
|
updateUrl.py |
|
|
|
|
urlWrapper.pl |
|
|
pipeline side |
Task Name: offLineTest
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
|
setup.csh |
|
|
|
start ConfigTables task |
test.pl |
|
|
|
Task Name: online
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
launch a run |
BeamTestLaunch.pl |
|
|
|
|
LicosLaunch.pl |
|
|
not used |
|
SVACLaunch.pl |
|
|
not used |
|
SVACWrapper.pl |
|
|
not used |
|
getAlgFile.pl |
|
|
not used |
Task Name: recon
Description:
The recon task is relatively complex. It splits a run into chunks, submits each one for reconstruction as a batch job (without GINO's knowledge), then merges the reconstructed chunks.
The first step, setupRecon, has almost all of the intelligence. It decides how many chunks to use and which events will go in which chunk. It writes a number of files which direct the action of later steps, including jobOptions files for reconstruction of the chunks.
doRecon controls the reconstruction of the chunks. It reads a list of jobs to run, and for each one, spawns a thread that submits a batch job, waits for it to complete, and returns its status. It then writes a file listing which, if any, of the chunks failed, and exits with an unsuccessful return code if there were failed chunks. When the TP first starts, it checks for this list of failed chunks, and if it is present and nonempty, it uses it instead of the original list of all chunks writted by setupRecon. This way, if some chunks fail, the TP can be rolled back and it will only need to redo the failed chunks. The failed chunk list is not registered as a pipeline dataset, since it violates logical constraints within GINO for a TP to modify its input this way.
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
finish up a run |
cleanup.py |
|
|
external side |
finish up a run |
cleanupWrapper.pl |
|
|
pipeline side |
control job for reconstruction of chunks |
doRecon.pl |
|
|
external side |
control job for reconstruction of chunks |
doReconWrapper.pl |
|
|
pipeline side |
|
genRTRLaunchWrapper.pl |
|
|
pipeline side |
make XML config file for task |
genXml.pl |
|
|
|
merge chunks of recon file |
mergeRecon.py |
|
|
external side |
merge chunks of recon file |
mergeReconWrapper.pl |
|
|
pipeline side |
|
recon.py |
|
|
obsolete |
reconstruct one chunk |
recon0ne.csh |
|
|
|
|
reconWrapper.pl |
|
|
obsolete |
|
reprocess-licos.csh |
|
|
delete |
|
reprocess-v3r1p5.csh |
|
|
delete |
|
reprocess-version.csh |
|
|
delete |
launch recon report |
RunRALaunchWrapper.pl |
|
|
pipeline side |
prepare chunk jobs |
setupRecon.py |
|
|
external side |
prepare chunk jobs |
setupReconWrapper.pl |
|
|
pipeline side |
Task Name: reconReport
Description: lo
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
make recon report |
genReconTestReport.pl |
|
|
external side |
make recon report |
genReconTestReportWrapper.pl |
|
|
pipeline side |
make XML file defining task |
genXml.pl |
|
|
|
Task Name: setup
Description:
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
setup for database access |
dbSetup8.cshrc |
|
|
Oracle 8, used by Python scripts |
setup for database access |
dbSetup10.cshrc |
|
|
Oracle 10, used by Perl scripts |
master setup file |
svacPlSetup.cshrc |
|
|
used by everything |
Task Name: svacTuple
Description: loads the database
Purpose |
Associated Scripts |
Input |
Output |
Comments |
|---|---|---|---|---|
make beamtest tuple |
RunRootAnalyzer.pl |
|
|
external side |
make beamtest tuple |
RunRootAnalyzerWrapper.pl |
|
|
pipeline side |
make XML file defining task |
genXml.pl |
|
|
|
|
reprocess-licos-v3r5p5.pl |
|
|
delete |
|
reprocess-v3r4p6.csh |
|
|
delete |
|
reprocess.csh |
|
|
delete |