What is the problem and why do we care?

  • At job startup, the pipeline interface is invoked with the command: bash pipeline_wrapper (see for example: /sdf/data/fermi/n/u41/L1/logs/DEV/AAA-testVerifyS3df-no-wrap/5.9/verify/000/slurm_pilot)
  • If we invoke the container with <job executable="${preamble} ${container_wrap} python --version" batchOptions="${extra}" /> (as in Brian's example), the pipeline interface defined in pipeline_wrapper is NOT available inside the container
  • This is a showstopper: L1Proc and other complex tasks use extensively the pipeline interface defined in pipeline_wrapper, to set/read pipeline variables, to create substreams, etc.

Possible solution (suggested by Brian on )

  • Write a lightweight interface to the pipeline environment, to obviate the need for invoking pipelineSet and pipelineCreateStream. Example here. 
  • Implemented in GPLTools as pipeline.py. This seems to work as of  

How we isolated and documented the issue

  • Task AAA-testVerifyS3df-wrapper (see xml here) loads the container with ${container_wrap} at line 50:

    • <job executable="${preamble} ${container_wrap} ${python27} ${L1ProcROOT}/verify.py" batchOptions="${extra}" />

  • Thereby losing access to the pipeline interface. For example, we try accessing pipelineSet and fail (full log here, error at line 179):
    • pipelineSet=
which: no pipelineSet in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)

  • Task AAA-testVerifyS3df-no-wrap (see xml here) defines the container, but does not invoke it at execution (line 50):

    • <job executable="${python27} ${L1ProcROOT}/verify.py" batchOptions="${extra}" />

    • (notably, the python errors in lines 173-213 of the logfile confirm that the container is not loaded correctly in the executable)
  • And still has access to the pipeline interface. For example, we try accessing pipelineSet and succeed (full log here, lines 221-225):
    • pipelineSet=
pipelineSet ()
{ 
    echo "Pipeline.$1: $2" >> ${PIPELINE_SUMMARY}
}
  • No labels