Table of Contents

Why use the pipeline to run Monte-Carlo jobs?

The pipeline offers a number of advantages when running large numbers of Monte-Carlo jobs:

Provides an easy way to submit a large number of jobs
Maintains record of all jobs run, including links to log files and other files produced during the job
Makes it easy to rerun any jobs that fail due to quirks in the SLAC batch system.
Makes it easy to register output datasets in the data catalog, which in turn makes it easy to keep track of what MC data is available.
Provides a web interface to allow the status of jobs to be monitored from anywhere.

Example Monte-Carlo task

Normally to run a set of Monte-Carlo jobs it is necessary to define a pipeline "Task". A task consists of an arbitrary graph of batch jobs and "scriptlets" to be run, however a typical Monte-Carlo task consists of just two steps:

...

Typically when generating large MC event samples it is necessary to run many MC jobs each of which generates some number of events. In the pipeline this is achieved by creating many streams within a given task. In the case of this example Monte-Carlo task each stream will run one batch job followed by one registration scriplet.

...

Running jobs

To create a pipeline task actually run some jobs it is necessary to write an XML configuration file. The key elements of the XML configuration file for the task above (with some details initially left out) are shown here:call the createStream function of the pipeline. The easiest way to do this is to use the pipeline createStream command when logged in to SLAC unix (e.g. noric).

No Format
~exodata/pipeline/prod/pipeline createStream [-options] <taskname> [file1 [file2 [...]]...]

where possible options include:

--stream <Stream ID=-1>	Integer stream identifier. Auto assigned if option not specified.
--nStreams <Number of Streams=1>	Number of streams to create, not valid if Stream ID specified
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="2ffa52aa-f4ce-43e2-9f99-836beb2f5556"><ac:plain-text-body><![CDATA[	--define <name=value>	Define a variable. Syntax is "name=value[,name2=value2,...]"	]]></ac:plain-text-body></ac:structured-macro>

For example to create 10 streams of the EXOMCBackground task defined above, overriding the default value of the MAXEVENTS variable we would use the following command:

No Format
~exodata/pipeline/prod/pipeline createStream ---define MAXEVENTS=100000 --nStreams 10 EXOMCBackground

Monitoring jobs using the pipeline web interface

The pipeline web interface can be access from the EXO data portal at:

http://exo-data.slac.stanford.edu/^{Image Added}

The web interface allows monitoring of the status of tasks and streams, viewing the log files of running or completed jobs and "rolling back" (rerunning) any failed jobs. It is also possible to view plots of how much CPU time jobs took, how much many jobs were running at a given time etc.

Also linked from the EXO data portal is the data catalog web interface. This allows the output files registered by the MC jobs to be viewed and downloaded.

Defining a task

To create a pipeline task it is necessary to write an XML configuration file. The key elements of the XML configuration file for the task above (with some details initially left out) are shown here:

Code Block

	xml
	xml


<?xml version="1.0" encoding="UTF-8"?>
<pipeline xmlns="http://glast-ground.slac.stanford.edu/pipeline"
          xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
          xs:schemaLocation="http://glast-ground.slac.stanford.

Code Block

xml


<?xml version="1.0" encoding="UTF-8"?>
<pipeline xmlns="http://glast-ground.slac.stanford.edu/pipeline"
          xmlns:xs="http://www.w3.org/2001/XMLSchema-instance"
          xs:schemaLocation="http://glast-ground.slac.stanford.edu/pipeline
          http://srs.slac.stanford.edu/Pipeline-II/schemas/2.0/pipeline.xsd">
  <task name="EXOMCBackground" type="EXO" version="1.3">
    <notation>A generic task for running EXO MC backgrounds</notation>
    <variables>
      <var name="EXODIR">/nfs/slac/g/exo</var>
      <var name="EXOBASE">${EXODIR}/software/builds/trunk</var>
      <var name="BATCHOPTIONS">-R &quot;select[-rhel30] rusage[scratch=1]&quot;</var>
      <var name="CORE_LIMIT">1024</var>
      <var name="MAXEVENTS">10000</var>
      <var name="PRINTMODULO">${MAXEVENTS/100}</var>
      <var name="MAXCPU">${MAXEVENTS/10}</var>
      <var name="MAXMEM">1000</var>
      <var name="OUTPUT_DIR">/nfs/slac/g/exo/exo_data/data/MC/backgrounds/test</var>
      <var name="OUTPUT_FORMAT">MC-background-%06d.root</var>
      <var name="OUTPUT_NAME">${format(pipeline.stream,OUTPUT_FORMAT)}</var>
      <var name="OUTPUT_FILE">${OUTPUT_DIR}/${OUTPUT_NAME}</var>
      <var name="DATACAT_DIR">EXO/Test</var>
      <var name="DATACAT_GROUP">MyGroup</var>
    </variables>

      <process name="runMonteCarlo">
         <job batchOptions="${BATCHOPTIONS}" maxCPU="${MAXCPU}" maxMemory="${MAXMEM}">
         ...
         </job>
      </process>


      <process name="register-ds">
         <notation>Register datasets created in this task</notation>
         <script>
         ...
         </script>
         <depends>
              <after process="runMonteCarlo"/>
         </depends>
     </process>
  </task>
</pipeline>

...

Code Block

	xml
	xml
title	EXOMCBackground.xml

<?xml version="1.0" encoding="UTF-8"?>
<pipeline xmlns="http://glast-ground.slac.stanford.edu/pipeline" 
          xmlns:xs="http://www.w3.org/2001/XMLSchema-instance" 
          xs:schemaLocation="http://glast-ground.slac.stanford.edu/pipeline 
          http://srs.slac.stanford.edu/Pipeline-II/schemas/2.0/pipeline.xsd">
  <task name="EXOMCBackground" type="EXO" version="1.5">
    <notation>A generic task for running EXO MC backgrounds</notation> 
    <variables>
      <var name="EXODIR">/nfs/slac/g/exo</var>
      <var name="EXOBASE">${EXODIR}/software/builds/trunk</var>
      <var name="BATCHOPTIONS">-R &quot;select[-rhel30] rusage[scratch=1]&quot;</var>
      <var name="CORE_LIMIT">1024</var>
      <var name="MAXEVENTS">10000</var> 
      <var name="PRINTMODULO">${MAXEVENTS/100}</var>
      <var name="INITIALSEED">pipeline.stream%100000</var>
      <var name="MAXCPU">${MAXEVENTS/10}</var>
      <var name="MAXMEM">1000</var>
      <var name="OUTPUT_DIR">/nfs/slac/g/exo/exo_data/data/MC/backgrounds/test</var>
      <var name="OUTPUT_FORMAT">MC-background-%06d.root</var>
      <var name="OUTPUT_NAME">${format(pipeline.stream,OUTPUT_FORMAT)}</var>
      <var name="OUTPUT_FILE">${OUTPUT_DIR}/${OUTPUT_NAME}</var>
      <var name="DATACAT_DIR">EXO/Test</var>
      <var name="DATACAT_GROUP">MyGroup</var>
    </variables>

      <process name="runMonteCarlo">
         <job batchOptions="${BATCHOPTIONS}" maxCPU="${MAXCPU}" maxMemory="${MAXMEM}"><![CDATA[
              ulimit -c ${CORE_LIMIT} # Limit core dumps
              set -e # exit on error

              # Create a scratch area to write the output to
              export SCRATCH_DIR=/scratch/exo/${PIPELINE_PROCESSINSTANCE}
              mkdir -p ${SCRATCH_DIR}
              gotEXIT()
              {
                 rm -rf ${SCRATCH_DIR}   
              }
              trap gotEXIT EXIT

              source ${EXOBASE}/setup.sh

              cat > background.exo <<EOF
              use exosim rec toutput
              /exosim/macro background.mac
              /exosim/filter true
              printmodulo ${PRINTMODULO}
              /exosim/initial_seed ${INITIALSEED}
              /exosim/run_number ${PIPELINE_STREAM}
              maxevents ${MAXEVENTS}
              /toutput/file ${SCRATCH_DIR}/output.root
              begin
              exit
              EOF

              cat > background.mac <<EOF
              /digitizer/wireNoise 800.000000
              /digitizer/APDNoise 2000.000000 
              /digitizer/LXeEnergyRes 0.015000
              /event/LXeEventsOnly true
              /event/digitizeWires true
              /event/digitizeAPDs true
              /gps/pos/type Volume
              /gps/pos/shape Cylinder
              /gps/pos/halfz 72.5 cm
              /gps/pos/radius 75.0 cm
              /gps/pos/centre 0.0 0.0 0.0 cm
              /gps/pos/confine HFE
              /gps/energy 0 keV
              /gps/particle ion
              /gps/ion 19 40 0 0
              /grdm/analogueMC 1
              EOF

              EXOAnalysis background.exo
              
              mkdir -p ${OUTPUT_DIR}
              cp -pv ${SCRATCH_DIR}/output.root ${OUTPUT_FILE}
              ]]>
         </job>
      </process>


      <process name="register-ds">
         <notation>Register datasets created in this task</notation>
         <script><![CDATA[ 
           attributes = {'sCreator':'tonyj','nThings':0.1}
           dsNew = datacatalog.newDataset(OUTPUT_NAME, "root", "EXOROOT", DATACAT_DIR, DATACAT_GROUP, "SLAC", OUTPUT_FILE)
           datacatalog.registerDataset(dsNew, attributes);
           ]]>
         </script>

         <depends>
              <after process="runMonteCarlo"/>
         </depends>

       </depends>

     </process>
  </task>
</pipeline>
</process>
  </task>
</pipeline>

Once a new task has been defined it can be uploaded using either the pipeline web interface (on the Admin page) or from the pipeline command on SLAC unix:

No Format
~exodata/pipeline/prod/pipeline load <xml-file>

Note that each file to be uploaded must have a unique task name and version number, so when uploading new version of a task it is necessary to increment the version number in the <task> element.

Child pages

Versions Compared

Old Version 4

New Version 5

Key

Why use the pipeline to run Monte-Carlo jobs?

Example Monte-Carlo task

Running jobs

Monitoring jobs using the pipeline web interface

Defining a task

Child pages

Page History

Versions Compared

Old Version 4

New Version 5

Key

Why use the pipeline to run Monte-Carlo jobs?

Example Monte-Carlo task

Running jobs

Monitoring jobs using the pipeline web interface

Defining a task