HPS MC Project Documentation

The hps-mc project provides a set of compiled tools and Python scripts for running MC jobs.

Installation

Prerequisites

The following prerequisites are required for running hps-mc:

• Maven (3.0 or greater)
• gcc (4.8 or greater)
• CMake (3.0 or greater)
• Python (2.7 or greater)
• SLIC
• ROOT (optional)

Bundled Dependencies

These dependencies are compiled and installed from source code within the project:

• egs5 event generator
• MadGraph 4 and MadGraph 5
• StdHep library and tools (based on version 5.6.1)

Installed Dependencies

The installation procedure will automatically download and install the following dependencies:

• LCIO (2.7.2)
• hps-java (HEAD of master)
• hps-fieldmaps

Building the Project

Start by checking out the project from github:

git clone https://github.com/jeffersonlab/hps-mc

Now, create a build dir and run CMake:

cd hps-mc; mkdir build; cd build
cmake -DCMAKE_INSTALL_PREFIX=../install ..

This will configure the project to be installed within a directory called install.

Next, run make to build the project:

make -j4 install

Job Environment

A number of environment variables are required for hps-mc to function properly.

These need to be setup by sourcing this script before attempting to run any jobs:

. hps-mc/install/bin/hps-mc-env.sh

The HPSJAVA_JAR variable will by default point to the copy built during the installation.  You can override this by setting it to point to a copy of the jar you want to use instead e.g.

export HPSJAVA_JAR=~/.m2/repository/org/hps/hps-distribution/4.0/hps-distribution-4.0-bin.jar

The SLIC application binary needs to be present in the environment (the SLIC environment is not managed directly by hps-mc), and you can check for this using:

which slic

If this application is not found then run the slic-env.sh to set it up before executing any hps-mc jobs.

ROOT is only currently used by one job script (tuple_job.py).  You need to set it up in your environment using the supplied thisroot.sh script for this job script to work.

Running Job Scripts

Running individual job scripts requires providing a JSON file with required parameters.

Running a job script looks like:

python job.py params.json

As an example, to generate WAB (Wide Angle Bremsstrahlung) events using MadGraph, these parameters could be used:

{                                                                          
    "job_num": 1,                                                                         
    "nevents": 10000,                                                                     
    "seed": 1234,                                                                         
    "output_dir": "output",                                                               
    "run_params": "1pt05",                                                                
    "output_files": {                                                                     
        "wab_events.lhe.gz": "wab_events_1234.lhe.gz"
    }                                                                                     
}     

The job_num specifies the job ID which is useful when running on a batch system.

The nevents parameter specifies how many events to generate.

The seed gives the random number seed for the job.

The output_dir is the destination directory for output files (it can be an absolute or relative path).

The run_params says what beam parameters to use.

The output_files section contains a list of source and destination files.  The file on the left side is copied to a file name listed on the right side.  (Here the file wab_events.lhe.gz will be copied to wab_events_1234.lhe.gz in the output directory.)

The names of the output file sources are dependent on the specific job script.  The destination can be any valid file name.

If you saved these job parameters as the file job.json you can run the WAB job as follows:

python hps-mc/python/jobs/wab_job.py job.json

This should create the file output/wab_events_1234.lhe.gz in the current directory.

You will probably want to run jobs locally in a scratch directory, as they will tend to write out many files!

Creating Job Workflows

In order to run jobs on a batch system such as LSF or Auger, the job parameters need to be expanded into a workflow, which is a JSON file containing parameters for all the individual jobs.

For instance, assume you have the following parameters in the file job.json:

{
    "nevents": 10000,
    "seed": 123456,
    "run_params": "1pt05",
    "detector": "HPS-EngRun2015-Nominal-v5-0-fieldmap",
    "run": 5772,
    "readout_steering": "/org/hps/steering/readout/EngineeringRun2015TrigPairs1_Pass2.lcsim",
    "recon_steering": "/org/hps/steering/recon/EngineeringRun2015FullReconMC.lcsim",
    "output_dir": "output",
    "output_files": {
        "tritrig.slcio": "tritrig_0001.slcio"
    }
}

This can be expanded into a workflow using the following command:

hps-mc-workflow -n 1000 -r 1234 -w tritrig hps-mc/python/jobs/tritrig_job.py job.json

Now you should see a local file called tritrig.json which contains information for running 1000 jobs of this type.

Running Jobs on the Batch System

Automatically submitting jobs to the batch system requires that you have created a workflow from your job parameters (covered in last section).

To submit all jobs in a workflow, execute a command similar to the following:

 hps-mc-batch -l $PWD/logs ./tritrig.json

You can also submit only certain job ids using a syntax like this to list specific job IDs:

hps-mc-batch -l $PWD/logs ./tritrig.json 1000 2000 [etc.]

Finally, it is possible to submit a range of job IDs:

hps-mc-batch -l $PWD/logs -r 0:99 ./tritrig.json

This will submit all the jobs IDs from 0 to 99 in the workflow.

Project Structure

The main project has the following directory structure:

Additionally, the following directory structure is installed to CMAKE_INSTALL_PREFIX.

The bin dir contains a large number of scripts and binaries that are created during the build process.