Search/Navigation:
Related:
SLAC
/EPP/HPS Public
Jefferson Lab/Hall B/HPS Run Wiki
S30XL-LESA/LDMX
Page History
...
Specific command line syntax is not standardized and will depend on what was implemented in that class by the author.
Steering Files
Steering File Locations
The standard location for steering files is steering-files/src/main/resources/org/hps/steering/ in HPS Java.
This folder is organized into the following sub-directories which contain sets of related steering files.
| Directory | Description |
|---|---|
| analysis | includes analysis template |
| broken | broken steering files (which may be removed soon!) |
| calibration | not really used currently |
| monitoring | steering files designed to run in the monitoring application |
| production | production steering including event filtering and Data Quality |
| readout | readout simulation to be run on MC data from SLIC |
| recon | production reconstruction |
| users | user files (organized into sub-directories by user name) |
These steering files can be accessed using their path on disk, or they can be referenced as classpath resources. The exact syntax depends on the command line tool.
For example, a steering file resource might be accessed like this using the job manager.
Command Line Tools
Job Manager
The JobManager runs a series of Drivers defined in an lcsim xml file on input events from one or more LCIO files. The XML steering file may contain lists of input file tags, or they may be (more typically) supplied with command line arguments. The job manager's class is listed in the distribution jar's manifest file, making it the default program which will run when using the java -jar command on the distribution.
Job Manager Command Line Usage
| No Format | ||
|---|---|---|
[1026 $] | ||
| Code Block | ||
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-3.6-SNAPSHOT-bin.jar -x /org/hps/steering/recon/EngineeringRun2015FullRecon.lcsim [...] |
The same steering file could also be accessed as a file from the local copy of HPS Java.
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar hps-java-trunk/steering-files/src/main/resources/org/hps/steering/recon/EngineeringRun2015FullRecon.lcsim [...] |
You can use a file rather than a resource if you running a steering file which is not checked into HPS Java or you are using a modified steering file that has not been packaged into the distribution jar.
Steering File Variables
For instance, suppose the XML file has this variable definition.
| Code Block | ||
|---|---|---|
| ||
<driver name="MyDriver" type="org.example.MyDriver">
<someNumber>${numVar}</someNumber>
</driver> |
The var would need to be resolved with a command like this.
| Code Block | ||
|---|---|---|
| ||
java -cp ./distribution/target/hps-distribution-bin.jar org.hps.evio.EvioToLcio -DnumVar=1234 [...] |
All variables defined in the XML steering files must be resolved from the command line or an error will occur.
System Properties
The command line tools may be affected by some Java system properties.
| Name | Description | Values |
|---|---|---|
| java.util.logging.config.file | Java logging config file | described in logging package doc |
| java.util.logging.config.class | Java logging config initialization class | described in logging package doc |
| hep.aida.IAnalysisFactory | AIDA backend factory class |
|
| disableSvtAlignmentConstants | Disables reading of SVT alignment constants from conditions db | true |
| org.hps.conditions.connection.file | Properties file with connection settings for conditions database | |
| org.hps.conditions.connection.resource | Resource that points to properties file with connection settings for conditions database |
|
These values are set as system properties in Java itself.
| Code Block | ||
|---|---|---|
| ||
java -DdisableSvtAlignmentConstants=true [...] |
When set in this way, these values are accessible as Java system properties at runtime.
Java Arguments
The JVM accepts a number of command line arguments that alter its behavior.
In particular, when running java you will likely want to increase the default heap space, as the default is too low for running HPS Java's full reconstruction.
| Code Block | ||
|---|---|---|
| ||
java -Xmx2g [...] |
That will change the heap space to 2 gigabytes.
Also, you may want to run the JVM in server mode.
| Code Block | ||
|---|---|---|
| ||
java -server [...] |
The server JVM has been optimized for performance speed and should run faster than the default client JVM.
Command Line Tools
Job Manager
The JobManager runs a series of Drivers defined in an lcsim xml file on input events from one or more LCIO files. The XML steering file may contain lists of input file tags, or they may be (more typically) supplied with command line arguments. The job manager's class is listed in the distribution jar's manifest file, making it the default program which will run when using the java -jar command on the distribution.
Job Manager Command Line Usage
Feb 18, 2016 4:02:27 PM org.lcsim.job.JobControlManager printHelp
INFO: java org.lcsim.job.JobControlManager [options] steeringFile.xml
usage:
-b,--batch Run in batch mode in which plots will not be
shown.
-D,--define <arg> Define a variable with form [name]=[value]
-d,--detector <arg> user supplied detector name (careful!)
-e,--event-print <arg> Event print interval
-h,--help Print help and exit
-i,--input-file <arg> Add an LCIO input file to process
-n,--nevents <arg> Set the max number of events to process
-p,--properties <arg> Load a properties file containing variable
definitions
-r,--resource Use a steering resource rather than a file
-R,--run <arg> user supplied run number (careful!)
-s,--skip <arg> Set the number of events to skip
-w,--rewrite <arg> Rewrite the XML file with variables resolved
-x,--dry-run Perform a dry run which does not process events |
Job Manager Command Line Options Table
| Switch | Description | Notes |
|---|---|---|
| -b | Activates batch mode plotting so plots will not be shown when running job. | This switch can be used to suppress plot display even if an AIDA plotter's show method is called from a driver. |
| -D | Add a variable definition that applies to the input steering file. | All variables in the XML input file must be resolved using this switch or an error will occur. |
| -d | Set the name of the detector model. | This should be a valid detector model defined in detector-data/detectors of HPS Java. |
| -e | Print out an informational message every N events. | Replaces EventMarkerDriver. When left out no information is printed about event numbers during the job. |
| -h | Print help and exit. | Using this with other arguments will still cause immediate help print and exit. |
| -i | Add an LCIO input file. | This switch can be used multiple times. |
| -n | Maximum number of events to run in job. | Leave this unset to run all events in the input files; internally -1 is used to indicate unlimited events. |
| -p | Load a properties file containing steering file variable definitions. | The input properties file should define name: value for each variable that needs to be defined. |
| -r | Treat the supplied steering as a classpath resource rather than a file. | These are typically resources from org/hps/steering in the steering-files module of HPS Java. |
| -R | Set the run number to be used when initializing the conditions system. | This has the side effect of "freezing" the conditions system, as the run numbers from the input events will be ignored. |
| -s | Skip N events at the beginning of the job. | |
| -w | Rewrite the XML steering file with the variables resolved. | This can be used as a simple template engine or for saving the values that were used for a job. |
| -x | Execute in dry run mode which means actual job will not execute. | Can be used to check for initialization errors. |
The steering file is supplied as an extra argument rather than with a command switch.
Here is an example using many of these command line arguments together. (This specific command will not work!)
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar -b -DoutputFile=output -d HPS-EngRun2015-Nominal-v3 -e 100 -i input.slcio \
-n 1000 -p myvars.prop -r -R 5772 -s 10 -w myjob.xml -x /org/hps/steering/dummy.lcsim |
If a detector name and run number are both supplied as arguments from the command line, the conditions system will be initialized and frozen, meaning that subsequent event numbers from data will be ignored.
EvioToLcio
The EvioToLcio tool converts EVIO to LCIO events and optionally can run a steering file job on the in-memory events. This allows the conversion and reconstruction to run in the same job/process for efficiency.
EvioToLcio Command Line Usage
| No Format |
|---|
[1037 $] java -cp ./distribution/target/hps-distribution-3.6-SNAPSHOT-bin.jar org.hps.evio.EvioToLcio
EvioToLcio [options] [evioFiles]
usage:
-b |
| No Format |
[1026 $] java -jar ./hps-java-trunk/distribution/target/hps-distribution-3.6-SNAPSHOT-bin.jar Feb 18, 2016 4:02:27 PM org.lcsim.job.JobControlManager printHelp INFO: java org.lcsim.job.JobControlManager [options] steeringFile.xml usage: -b,--batch Run in batch mode in which plots will not be shown. -D,--define <arg> Define a variable with form [name]=[value] -d,--detector <arg> user supplied detector name (careful!) -e,--event-print <arg> Event print interval -h,--help Print help and exit -i,--input-file <arg> Add an LCIO input file to process -n,--nevents <arg> Set the max number of events to process -p,--properties <arg> Load a properties file containing variable definitions -r,--resource Use a steering resource rather than a file -R,--run <arg> user supplied run number (careful!) -s,--skip <arg> Set the number of events to skip -w,--rewrite <arg> Rewrite the XML file with variables resolved -x,--dry-run Performenable aheadless drymode runin which does not process events |
Job Manager Command Line Options Table
| Switch | Description | Notes |
|---|---|---|
| -b | Activates batch mode plotting so plots will not be shown when running job. | This switch can be used to suppress plot display even if an AIDA plotter's show method is called from a driver. |
| -D | Add a variable definition that applies to the input steering file. | All variables in the XML input file must be resolved using this switch or an error will occur. |
| -d | Set the name of the detector model. | This should be a valid detector model defined in detector-data/detectors of HPS Java. |
| -e | Print out an informational message every N events. | Replaces EventMarkerDriver. When left out no information is printed about event numbers during the job. |
| -h | Print help and exit. | Using this with other arguments will still cause immediate help print and exit. |
| -i | Add an LCIO input file. | This switch can be used multiple times. |
| -n | Maximum number of events to run in job. | Leave this unset to run all events in the input files; internally -1 is used to indicate unlimited events. |
| -p | Load a properties file containing steering file variable definitions. | The input properties file should define name: value for each variable that needs to be defined. |
| -r | Treat the supplied steering as a classpath resource rather than a file. | These are typically resources from org/hps/steering in the steering-files module of HPS Java. |
| -R | Set the run number to be used when initializing the conditions system. | This has the side effect of "freezing" the conditions system, as the run numbers from the input events will be ignored. |
| -s | Skip N events at the beginning of the job. | |
| -w | Rewrite the XML steering file with the variables resolved. | This can be used as a simple template engine or for saving the values that were used for a job. |
| -x | Execute in dry run mode which means actual job will not execute. | Can be used to check for initialization errors. |
The steering file is supplied as an extra argument rather than with a command switch.
Here is an example using many of these command line arguments together. (This specific command will not work!)
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar -b -DoutputFile=output -d HPS-EngRun2015-Nominal-v3 -e 100 -i input.slcio \
-n 1000 -p myvars.prop -r -R 5772 -s 10 -w myjob.xml -x /org/hps/steering/dummy.lcsim |
If a detector name and run number are both supplied as arguments from the command line, the conditions system will be initialized and frozen, meaning that subsequent event numbers from data will be ignored.
EvioToLcio
The EvioToLcio tool converts EVIO to LCIO events and optionally can run a steering file job on the in-memory events. This allows the conversion and reconstruction to run in the same job/process for efficiency.
EvioToLcio Command Line Usage
| No Format |
|---|
[1037 $] java -cp ./distribution/target/hps-distribution-3.6-SNAPSHOT-bin.jar org.hps.evio.EvioToLcio
EvioToLcio [options] [evioFiles]
usage:
-b enable headless mode in which plots will not show
-d <arg> detector name (required)
-D <arg> define a steering file variable with format -Dname=value
-f <arg> text file containing a list of EVIO files
-h print help and exit
-L <arg> log level (INFO, FINE, etc.)
-l <arg> path of output LCIO file
-m <arg> set the max event buffer size
-M use memory mapping instead of sequential reading
-n <arg> maximum number of events to process in the job
-r interpret steering from -x argument as a resource instead of a
file
-R <arg> fixed run number which will override run numbers of input
files
-t <arg> specify a conditions tag to use
-v print EVIO XML for each event
-x <arg> LCSim steeering file for processing the LCIO events |
EvioToLcio Command Line Options Table
| Switch | Description | Notes |
|---|---|---|
| -b | Activates batch mode plotting so plots will not be shown when running job. | |
| -d | Set the name of the detector model. | |
| -D | Add a variable definition that applies to the input steering file. | |
| -f | Text file containing list of input EVIO files. | |
| -L | Set log level. | DEPRECATED Use logging config file or class instead. |
| -l | Path of output LCIO file. | |
| -m | Set the max event buffer size. | Experts only. |
| -M | Use memory mapping in EVIO reader instead of sequential access. | Experts only. |
| -n | Maximum number of events to process in the job. | |
| -R | Set the run number to be used when initializing the conditions system. | |
| -t | Specify a conditions tag for filtering conditions records. | |
| -v | Print out EVIO converted to XML for every event. | For verbose debugging of events. |
| -r | Interpret steering file from -x as a classpath resource rather than a file. | |
| -x | Steering file | Could be resource or file depending on if -r switch is used. |
Here is an example showing how to use most of these command line options.
plots will not show
-d <arg> detector name (required)
-D <arg> define a steering file variable with format -Dname=value
-f <arg> text file containing a list of EVIO files
-h print help and exit
-L <arg> log level (INFO, FINE, etc.)
-l <arg> path of output LCIO file
-m <arg> set the max event buffer size
-M use memory mapping instead of sequential reading
-n <arg> maximum number of events to process in the job
-r interpret steering from -x argument as a resource instead of a
file
-R <arg> fixed run number which will override run numbers of input
files
-t <arg> specify a conditions tag to use
-v print EVIO XML for each event
-x <arg> LCSim steeering file for processing the LCIO events |
EvioToLcio Command Line Options Table
| Switch | Description | Notes |
|---|---|---|
| -b | Activates batch mode plotting so plots will not be shown when running job. | |
| -d | Set the name of the detector model. | |
| -D | Add a variable definition that applies to the input steering file. | |
| -f | Text file containing list of input EVIO files. | |
| -L | Set log level. | DEPRECATED Use logging config file or class instead. |
| -l | Path of output LCIO file. | |
| -m | Set the max event buffer size. | Experts only. |
| -M | Use memory mapping in EVIO reader instead of sequential access. | Experts only. |
| -n | Maximum number of events to process in the job. | |
| -R | Set the run number to be used when initializing the conditions system. | |
| -t | Specify a conditions tag for filtering conditions records. | |
| -v | Print out EVIO converted to XML for every event. | For verbose debugging of events. |
| -r | Interpret steering file from -x as a classpath resource rather than a file. | |
| -x | Steering file | Could be resource or file depending on if -r switch is used. |
Here is an example showing how to use most of these command line options.
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar -b -DoutputFile=output -f evio_files.txt -l lcio_file_output -m 50 \
-v -M -n 1000 -d HPS-EngRun2015-Nominal-v3 -R 5772 -t pass1 -r -x /org/hps/steering/dummy.lcsim |
Some of these arguments are similar to the job manager, but the steering file is supplied in a different way. Evio2Lcio uses a command switch to specifiy the steering whereas the job manager expects this as an extra argument without a command switch.
Run Scripts
Run scripts that wrap a number of HPS Java command line utilities are generated when building the distribution.
After the build completes, they should be found in this HPS Java directory.
| No Format |
|---|
distribution/target/appassembler/bin/ |
For instance, the EvioToLcio utility can be run using this script.
| Code Block | ||
|---|---|---|
| ||
distribution/target/appassembler/bin/evio2lcio.sh [...] |
These scripts have several advantages over running the java commands yourself.
- A full classpath is created in the script so it is not necessary to use the distribution jar.
- Reasonable JVM options are set such as the min heap size.
- Logging is configured automatically by a default logging properties file.
- Less typing of Java boilerplate commands ("java -jar" etc.).
- You do not need to know the corresponding class's full name to run its command line utility.
Using symlinks to these scripts works fine e.g.
| Code Block | ||
|---|---|---|
| ||
ln -s distribution/target/appassembler/bin/evio2lcio.sh
./evio2lcio.sh |
When using these scripts, you cannot directly supply Java system properties, so the JAVA_OPTS variable should be used instead.
| Code Block | ||
|---|---|---|
| ||
export JAVA_OPTS="-DdisableSvtAlignmentConstants=true" |
The full list of Java system properties to be used should be included in this variable. You should not set the options -Xmx or -Djava.util.logging.config.class, as these are set by each script.
Steering Files
Steering File Locations
The standard location for steering files is steering-files/src/main/resources/org/hps/steering/ in HPS Java.
This folder is organized into the following sub-directories which contain sets of related steering files.
| Directory | Description |
|---|---|
| analysis | includes analysis template |
| broken | broken steering files (which may be removed soon!) |
| calibration | not really used currently |
| monitoring | steering files designed to run in the monitoring application |
| production | production steering including event filtering and Data Quality |
| readout | readout simulation to be run on MC data from SLIC |
| recon | production reconstruction |
| users | user files (organized into sub-directories by user name) |
These steering files can be accessed using their path on disk, or they can be referenced as classpath resources. The exact syntax depends on the command line tool.
For example, a steering file resource might be accessed like this using the job manager.
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar -x /org/hps/steering/recon/EngineeringRun2015FullRecon.lcsim [...] |
The same steering file could also be accessed as a file from the local copy of HPS Java.
| Code Block | ||
|---|---|---|
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar hps-java-trunk/steering-files/src/main/resources/org/hps/steering/recon/EngineeringRun2015FullRecon.lcsim [...] |
You can use a file rather than a resource if you running a steering file which is not checked into HPS Java or you are using a modified steering file that has not been packaged into the distribution jar.
Steering File Variables
For instance, suppose the XML file has this variable definition.
| Code Block | ||
|---|---|---|
| ||
<driver name="MyDriver" type="org.example.MyDriver">
<someNumber>${numVar}</someNumber>
</driver> |
The var would need to be resolved with a command like this.
| Code Block | ||
|---|---|---|
| ||
java -cp . | ||
| Code Block | ||
| ||
java -jar ./hps-java-trunk/distribution/target/hps-distribution-bin.jar -b -DoutputFile=output -f evio_files.txt -l lcio_file_output -m 50 \ -v -M -n 1000 -d HPS-EngRun2015-Nominal-v3 -R 5772 -t pass1 -r -x /org/hps/steering/dummy.lcsim |
Some of these arguments are similar to the job manager, but the steering file is supplied in a different way. Evio2Lcio uses a command switch to specifiy the steering whereas the job manager expects this as an extra argument without a command switch.
Run Scripts
Run scripts that wrap a number of HPS Java command line utilities are generated when building the distribution.
After the build completes, they should be found in this HPS Java directory.
| No Format |
|---|
distribution/target/appassembler/bin/ |
For instance, the EvioToLcio utility can be run using this script.
| Code Block | ||
|---|---|---|
| ||
distribution/target/appassembler/bin/evio2lcio.sh [...] |
These scripts have several advantages over running the java commands yourself.
- A full classpath is created in the script so it is not necessary to use the distribution jar.
- Reasonable JVM options are set such as the min heap size.
- Logging is configured automatically by a default logging properties file.
- Less typing of Java boilerplate commands ("java -jar" etc.).
- You do not need to know the corresponding class's full name to run its command line utility.
org.hps.evio.EvioToLcio -DnumVar=1234 [...] |
All variables defined in the XML steering files must be resolved from the command line or an error will occur.
System Properties
The command line tools may be affected by some Java system properties.
| Name | Description | Values |
|---|---|---|
| java.util.logging.config.file | Java logging config file | described in logging package doc |
| java.util.logging.config.class | Java logging config initialization class | described in logging package doc |
| hep.aida.IAnalysisFactory | AIDA backend factory class |
|
| disableSvtAlignmentConstants | Disables reading of SVT alignment constants from conditions db | true |
| org.hps.conditions.connection.file | Properties file with connection settings for conditions database | |
| org.hps.conditions.connection.resource | Resource that points to properties file with connection settings for conditions database |
|
These values are set as system properties in Java itselfUsing symlinks to these scripts works fine e.g.
| Code Block | ||
|---|---|---|
| ||
lnjava -s distribution/target/appassembler/bin/evio2lcio.sh ./evio2lcio.shDdisableSvtAlignmentConstants=true [...] |
When set in this way, these values are accessible as Java system properties at runtime.
Java Arguments
The JVM accepts a number of command line arguments that alter its behavior.
In particular, when running java you will likely want to increase the default heap space, as the default is too low for running HPS Java's full reconstructionWhen using these scripts, you cannot directly supply Java system properties, so the JAVA_OPTS variable should be used instead.
| Code Block | ||
|---|---|---|
| ||
exportjava JAVA_OPTS="-DdisableSvtAlignmentConstants=true"-Xmx2g [...] |
That will change the heap space to 2 gigabytes.
Also, you may want to run the JVM in server mode.
| Code Block | ||
|---|---|---|
| ||
java -server [...] |
The server JVM has been optimized for performance speed and should run faster than the default client JVMThe full list of Java system properties to be used should be included in this variable. You should not set the options -Xmx or -Djava.util.logging.config.class, as these are set by each script.
Logging and Debugging
Logging Config
...
Overview
Content Tools