You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Next »

The SLAC workflow engine has been extended to allow multiple workflow engine jobs to run inside a "pilot" job. The main intent is to allow multiple (possible single thread) jobs to run on hosts supporting many threads, for example on CORI at NERSC, although the implementation is not specific to any particular machine.

Implementation

The workflow engine works by having "JobDaemons" for each type of batch system (LSF, Condor, SLURM etc). In the case of pilot jobs we use a special daemon called the PilotJobDaemon, which does not submit jobs to a batch system, but instead keeps the jobs in an internal queue until a pilot job comes and pulls them out of the queue to execute them. For the most part, using a PilotJobDaemon is transparent to the workflow engine, and the developer of specific workflows.

For DESC we have set up two JobDaemons at NERSC, both running under the shared "desc" account:

  • SlurmJobDaemon which submits jobs to SLURM. The "jobsite" which must be used to submit jobs to this daemon  is "CORI".
  • PilotJobDaemon which queues up jobs until a pilot job fetches them and runs them. The "jobsite" which must be used to submit jobs to this daemon  is "CORIP".

To a first approximation, the only change to the workflow to switch from one to the other is to change the jobsite. Since the jobsite can be set dynamically it is possible to have one workflow which submits jobs to either site (or even both sites).

Batch Options

To the extent possible the options supported by the PilotJobDaemon are the same as those supported by the SlurmJobDaemon. Details below:

OptionAliasDefaultMeaningComments
-N--nodes1The number of nodes on which the job will runOnly for compatibility with SLURM. Option is ignored.
-t--time01:00:00The wallclock time allowed for the jobThis is used for scheduling jobs in the pilot, but is not currently enforced by the pilot.
-L--licensenoneThe list of licenses required by the job separated by commas, e.g. -L SCRATCHAccepted but not yet used.
-C--constraintnoneThe list of constraints required by the job, separated by commas, e.g. -C haswellAccepted but not yet used.
-p--partitionnoneThe partition in which the job will be run.Only for compatibility with SLURM. Option is ignored.
-c--cpus-per-task1The number of cpus (threads) which will be allocated to this job.This is used for scheduling jobs in the pilot, but is not currently enforced by the pilot.
 --ntasks-per-node1 Only for compatibility with SLURM. Option is ignored.
-J--job_name The name of the job.Only for compatibility with SLURM. Option is ignored.

In addition the memory and maxcpu can be specified as part of the workflow job definition (in XML). These are used for scheduling the job in the pilot, but are not currently enforced by the pilot.

Pilot Jobs

In the current implementation the pilot jobs are not submitted automatically, although this may change in future. Currently to submit the default pilot job simply login as user "desc" (separate instructions needed?) and run the following:

sbatch ~/jobcontrol/runPilot.sh

It is possible to customize the pilot for specific applications, for example you might want to setup some specific environment which will be inherited by all jobs running in the pilot, or you might want to customize some of the options supported by the job daemon. In either case you can create your own modified copy of the runPilot.sh script, and/or specify an option on the command line.

The options supported by the JobControlPilot are:

-c N : The total number of cores of share among all running jobs (default: 64)
-h VAL : The host from which this pilot will attempt to pull jobs (default: corigrid.nersc.gov)
-i N : The time after which the pilot will die if no work is provided (seconds) (default: 300)
-m N : The total memory (in kB) of this machine to share among all running jobs (default: 64000000)
-o : True if OK to overwrite existing files (default: false)
-p N : The port that the pilot will attempt to pull jobs from (default: 0)
-r N : The maximum runtime for the job (seconds) (default: 172800)
-s VAL : The service name of the pilot service (default: PilotJobProvider)
-u VAL : The user name under which the pilot service is running (default: desc)

Any number of pilot jobs can be submitted.

 

  • No labels