Versions Compared

Old Version 16

changes.mady.by.user Unknown User (mattlangston)

Saved on

compared with

New Version Current

changes.mady.by.user Unknown User (mattlangston)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Migration of unmigrated content due to installation of a new plugin

Introduction

This is proposed file format for the xml file upload utility. I took the existing recon-EM2-v1r0 pipeline as a model. I haven't fully documented the format yet, nor have I created a schema for it yet. Both of these are on the TODO list. However, for those of you creating pipelines I believe this should make sense at some levelbut there is an XML shema you can reference at http://glast-ground.slac.stanford.edu/pipeline/pipeline-v0.xsd. The pages you are reading are the primary documentation for the pipeline configuration, and these pages will be updated as new documentation becomes available.

Defining your pipeline

Here are the steps you should take to create an xml configuration file to define a pipeline:

  1. The top-level root element in the configuration file is a pipeline. Define the type attribute, and the base-path and run-log-path elements. The type attribute must be from a finite set that we define which are listed below.
  2. Define the list of executables your pipeline will run in the executable-list section. This section simply gives a friendly name and a version string (which is used for documentation purposes only) to the long file paths of the executables your pipeline will run. You refer to these executables by their friendly names in the processing-step elements defined below in step 5.
  3. Define the batch job configurations your pipeline needs in the batch-job-configuration-list section. This section simply gives a friendly name to a batch queue, a working directory and the directory where the log file will go. You refer to these batch job configurations by their friendly names in the processing-step elements defined below in step 5. The queue attribute must be from a finite set that we define which are listed below. The working-directory and log-file-path elements are directories that are relative to base-path defined in step 1.
  4. Define the list of files your pipeline is concerned with in the file-list section. This section simply gives friendly names to long files paths and defines some meta-information about the files. You refer to these files by their friendly names in the input-file-list and output-file-list sections in the processing-step elements defined below in step 5. The type and file-type attributes must be from a finite set that we define which are listed below.
  5. Finally, define the processing steps of your pipeline in the processing-step-list section. Each processing-step in this section refers to the executable defined in step 2 (via the executable attribute), the batch job configuration defined in step 3 (via the batch-job-configuration attribute), and the list of input files and output files (which you defined in step 4) in the input-file-list and output-file-list sections, respectively.

Example Configuration File

Here is an example pipeline configuration file for the recon-EM2-v1r0 pipeline.

Code Block
xml
xml
<?xml version="1.0" encoding="UTF-8"?>
<pipeline
  name="recon-EM2-v1r0"
  type="Reconstruction"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="http://glast-ground.slac.stanford.edu/pipeline/pipeline-v0.xsd">
	
  <base-path>/nfs/farm/g/glast/u12/EM2</base-path>
	  <run-log-path>/temp/</run-log-path>

	  <executable-list>
		
    <executable name="reconWrapper" version="v1r0">
      /nfs/slac/g/svac/common/pipeline/EM2/svacPipeline/recon/v1r0/reconWrapper.pl
    </executable>
		
    <executable name="RunRALaunchWrapper" version="v1r0">
      /nfs/slac/g/svac/common/pipeline/EM2/svacPipeline/recon/v1r0/RunRALaunchWrapper.pl
    </executable>
		

    <executable name="genRTRLaunchWrapper" version="v1r0">
      /nfs/slac/g/svac/common/pipeline/EM2/svacPipeline/recon/v1r0/genRTRLaunchWrapper.pl
    </executable>
		
    <executable name="urlWrapper" version="v1r0">
      /nfs/slac/g/svac/common/pipeline/EM2/svacPipeline/lib/urlWrapper.pl
    </executable>
	
  </executable-list>

	  <batch-job-configuration-list>
		
    <batch-job-configuration name="long-job" queue="long">
			      <working-directory>log/$(RUN_NAME)/</working-directory>
			      <log-file-path>rootData/$(RUN_NAME)/calib-v1r0/grRoot</log-file-path>
		    </batch-job-configuration>
		
    <batch-job-configuration name="short-job" queue="short">
			      <working-directory>log/$(RUN_NAME)/</working-directory>
			      <log-file-path>rootData/$(RUN_NAME)/calib-v1r0/grRoot</log-file-path>
		    </batch-job-configuration>
	
  </batch-job-configuration-list>
	
  <file-list>
		    <file name="digi"       type="DIGI"   file-type="root">rootData/$(RUN_NAME)/grRoot</file>
		    <file name="jobOptions" type="text"   file-type="jobOpt">rootData/$(RUN_NAME)/calib-v1r0/grRoot</file>
		    <file name="merit"      type="merit"  file-type="root">rootData/$(RUN_NAME)/calib-v1r0/grRoot</file>
		    <file name="recon"      type="RECON"  file-type="root">rootData/$(RUN_NAME)/calib-v1r0/grRoot</file>
		    <file name="script"     type="script" file-type="csh">rootData/$(RUN_NAME)/calib-v1r0/grRoot</file>
	  </file-list>
	
  <processing-step-list>
		

    <processing-step name="recon" executable="reconWrapper" batch-job-configuration="long-job">
			      <input-file-list>
				        <file ref="digi"/>
			      </input-file-list>
			          <output-file-list>
				        <file ref="jobOptions"/>
				        <file ref="merit"/>
				        <file ref="recon"/>
				        <file ref="script"/>
			      </output-file-list>
		    </processing-step>
		
    <processing-step name="LaunchSVAC" executable="RunRALaunchWrapper" batch-job-configuration="short-job">
			      <input-file-list>
				        <file ref="digi"/>
				        <file ref="recon"/>
			      </input-file-list>
		    </processing-step>
		
    <processing-step name="LaunchReport" executable="genRTRLaunchWrapper" batch-job-configuration="short-job">
			      <input-file-list>
				        <file ref="digi"/>
				        <file ref="recon"/>
			      </input-file-list>
		    </processing-step>
		

    <processing-step name="reconRootFile" executable="urlWrapper" batch-job-configuration="short-job">
			      <input-file-list>
				        <file ref="recon"/>
			      </input-file-list>
		    </processing-step>
		

    <processing-step name="meritRootFile" executable="urlWrapper" batch-job-configuration="short-job">
			      <input-file-list>
				        <file ref="merit"/>
			      </input-file-list>
		    </processing-step>
	
  </processing-step-list>

</pipeline>

So, take a look at the example and see if it makes sense. Let me know if you have questions, or if you see anything that should be changed.
I still have some work to do on getting the data to import properly, but you can start creating these xml configuration files for your pipelines.
Expect there to be some initial problems, as this has so far only been tested by me.
Warmest regards, Matt

Values for Elements used in Configuration File

Possible Values for Pipeline Type

SimReconDigi

test

Analysis

Reconstruction

Digitization

Report

SystemTest

Simulation

Reprocessing

Conversion

...