Confluence will be down for maintenance June 14 2024 at 6AM PT.
A top level definition of work to be performed by the pipeline. A task may consist of one or more processes, and zero or more nested subtasks. Tasks are defined by an XML file.
A stream represents a single request to run the processes within a task. Streams always have an associated stream number which must be unique within each task. The stream number is always set at create stream time, either explicitly by the user or impl.
A task contained within a parent task.
A stream corresponding to a sub-task.
A single step within a task or subtask. Each process must be either a script or a job.
A process which results in a batch job being run.
A process which results in a script being run inside the pipeline server itself. These small scripts are typically used to perform simple calculations, set variables, create subtasks, or make entries in the data catalog. Scripts can call functions provided by the pipeline itself, as well as additional functions for adding entries to the data catalog.
Pipeline variables can be defined in a pipeline XML file, either at the task level, or at the level of individual processes. They can also be defined at create stream time. Processes inherit variables from
Variables from other processes or tasks can also be accessed using the pipeline object.
Batch jobs will always have the following environment variables set:
Variable |
Usage |
---|---|
PIPELINE_PROCESSINSTANCE |
The internal database id of the process instance |
PIPELINE_STREAM |
The stream number |
PIPELINE_STREAMPATH |
The stream path. For a top level task this will be the same as the stream number, for sub-tasks this will be of the form i.j.k |
PIPELINE_TASK |
The task name |
PIPELINE_PROCESS |
The process name |
To get details on using the pipeline II client try
~glast/pipeline-II/pipeline help
Which currently gives:
Syntax: pipeline <command> <args> where command is one of: createStream <task> <stream> <env> where <task> is the name of the task to create (including optional version number) <stream> is the stream number to create. <env> are environment variables to pass in, of the form var=value\{,var=value...\}
~glast/pipeline-II/pipeline createStream CHS-level1 2 "downlinkID=060630001,numChunks=10,taskBase=/nfs/farm/g/glast/u23/ISOC-devel/Pipelines/CHS-level1,productVer=0,fastCopy=0"