Page History

...

Pipeline tasks are structured as a linear chain of steps.  Each step must succeed before going on to the next one.  Each step must be attempted, until one fails or the last one succeeds.  That is to say, there is no flow control, and no parallelism.  In order to deal with this, and to reduce the chance that a step would not be reached due to the failure of a previous step on which it did not logically depend, the work we do in the pipeline is split into 7 tasks, all but one of which (updateELogDB) are launched internallyby other tasks.  This launching is done without GINO's (the pipeline infrastructure) knowledge.

Most steps within a task (Task Processes (TPs)) consist of a wrapper, which interfaces with the pipeline infrastructure ( GINO ) and a script which does the work, usually by running some external app.

...

Task Name: recon

Description: 

The recon task is relatively complex.  It splits a run into chunks, submits each one for reconstruction as a batch job (without GINO's knowledge), then merges the reconstructed chunks.

The first step, setupRecon, has almost all of the intelligence.  It decides how many chunks to use and which events will go in which chunk.  It writes a number of files which direct the action of later steps, including jobOptions files for reconstruction of the chunks.

doRecon controls the reconstruction of the chunks.  It reads a list of jobs to run, and for each one, spawns a thread that submits a batch job, waits for it to complete, and returns its status.  It then writes a file listing which, if any, of the chunks failed, and exits with an unsuccessful return code if there were failed chunks.  When the TP first starts, it checks for this list of failed chunks, and if it is present and nonempty, it uses it instead of the original list of all chunks writted by setupRecon.  This way, if some chunks fail, the TP can be rolled back and it will only need to redo the failed chunks.  The failed chunk list is not registered as a pipeline dataset, since it violates logical constraints within GINO for a TP to modify its input this way.

...