Page History

...

You will probably want to run jobs locally in a scratch directory, as they will tend to write out many files!

Creating Job Workflows

In order to run jobs on a batch system such as LSF or Auger, the job parameters need to be expanded into a workflow, which is a JSON file containing parameters for all the individual jobs.

For instance, assume you have the following parameters in the file job.json:

Input and Output Files

The job parameters may specify input files if the script uses them and optionally output file locations.

{                                                                          
    "input_files": {
        "events1.stdhep": "/path/to/events1.stdhep",
        "events2.stdhep": "/path/to/events2.stdhep",
    } 
    "output_files": {                      
        "events1.slcio": "my_events1.slcio",
        "events2.slcio": "my_events2.slcio"
    },
    "output_dir": "/path/to/outdir"                                                                                  
}     

In the above toy example, the files listed in input_files will be copied from the absolute path on the right to the file name on the left (format is "destination": "source").

The output files will be copied from the left hand path in the local scratch dir to the file name on the right (format is "source": "destination").

All output files will be copied to the directory listed under "output_dir", which can be an absolute or relative path.

To keep the names of the output files created by the job, simply list the same file name for the output file entries.

{                                                                            
    "output_files": {                      
        "events1.slcio": "events1.slcio",
        "events2.slcio": "events2.slcio"
    }
}     

This will make the job copy the file events1.slcio to a file with the same name in the output directory.

Creating Job Workflows

In order to run jobs on a batch system such as LSF or Auger, the job parameters need to be expanded into a workflow, which is a JSON file containing parameters for all the individual jobs.

For instance, assume you have the following parameters in the file job.json:

{
    "nevents": 10000,
    "seed": 123456,
    "run_params": "1pt05",
    "detector": "HPS-EngRun2015-Nominal-v5-0-fieldmap",
    "run": 5772,
    "readout_steering": "/org/hps/steering/readout/EngineeringRun2015TrigPairs1_Pass2.lcsim",
    "recon_steering": "/org/hps/steering/recon/EngineeringRun2015FullReconMC.lcsim",
    "output_dir": "output",
    "output_files": {
        "tritrig.slcio": "tritrig_0001.slcio"
    }
}

This can be expanded into a workflow using the following command:

hps-mc-workflow -n 1000 -r 1234 -w tritrig hps-mc/python/jobs/tritrig_job.py job.json

Now you should see a local file called tritrig.json which contains information for running 1000 jobs of this type.

The input files for a workflow may be supplied in one of two ways.

A file glob will supply multiple input files to the workflow, one per job.

"input_files" : {
    "beam.stdhep": "/not/a/real/path/beam*.stdhep"
}

Mutiple files can be supplied based on the following syntax.

"input_files" : {
    "beam.stdhep

{
    "nevents": 10000,
    "seed": 123456,
    "run_params": "1pt05",
    "detector": "HPS-EngRun2015-Nominal-v5-0-fieldmap",
    "run": 5772,
    "readout_steering": "/org/hps/steering/readout/EngineeringRun2015TrigPairs1_Pass2.lcsim",
    "recon_steering": "/org/hps/steering/recon/EngineeringRun2015FullReconMC.lcsim",
    "output_dir": "output",
    "output_files": {
        "tritrig.slcio": "tritrig_0001.slcio"/not/a/real/path/beam*.stdhep": 10
    }
}

This can be expanded into a workflow using the following command:

hps-mc-workflow -n 1000 -r 1234 -w tritrig hps-mc/python/jobs/tritrig_job.py job.json

Now you should see a local file called tritrig.json which contains information for running 1000 jobs of this typewill expand into JSON parameters that include 10 files per job in the workflow.

Running Jobs on the Batch System

Automatically submitting jobs to the batch system requires that you have created a workflow from your job parameters (covered in last section)last section).

The following commands use the script hps-mc-bsub to submit jobs to LSF (e.g. SLAC environment).

The command hps-mc-jsub should be used instead when submitting at JLab.

To submit all jobs in a workflow, execute a command similar to the following:

 hps-mc-batchbsub -l $PWD/logs ./tritrig.json

...

hps-mc-batchbsub -l $PWD/logs ./tritrig.json 1000 2000 [etc.]

...

hps-mc-batchbsub -l $PWD/logs -r 0:99 ./tritrig.json

...

# generate A-prime events using Madgraph 4
ap = MG4(name="ap",
        run_card="run_card_"+params.run_params+".dat",
        params={"APMASS": params.apmass},
        outputs=[filename],
        nevents=params.nevents)

Finally, the components should be added to the job and the job should be run.

job.components = [ap]
job.run()

The specific way that input and output files are used depends on the job script.

...