Versions Compared

Old Version 9

changes.mady.by.user Maria Elena Monzani

Saved on

compared with

New Version Current

changes.mady.by.user Maria Elena Monzani

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

  • At job startup, the pipeline interface is invoked with the command: bash pipeline_wrapper (see for example: /sdf/data/fermi/n/u41/L1/logs/DEV/AAA-testVerifyS3df-no-wrap/5.9/verify/000/slurm_pilot)
  • If we invoke the container with <job executable="${preamble} ${container_wrap} python --version"batchOptions="${extra}" /> (as in Brian's example), the pipeline interface defined in pipeline_wrapper is NOT available inside the container
  • This is a showstopper: L1Proc and other complex tasks use extensively the pipeline interface defined in pipeline_wrapper, to set/read pipeline variables, to create substreams, etc.

Possible solution (suggested by Brian on )

  • Write a lightweight interface to the pipeline environment, to obviate the need for invoking pipelineSet and pipelineCreateStream. Example here. 
  • Implemented in GPLTools as pipeline.py. This seems to work as of  

How we isolated and documented the issue

...