You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 12 Current »

Accounts and Resources

A description of the LCLS computing resources can be found at the PCDS Computing page.

Setting Up the Software Environment

For the analysis software to work correctly it has to run in a correct environment. LCLS Analysis environment consists of few environment variables that either point to a location of specific directories or define some configuration options such as type of the machine, compiler versions, etc. Users do not need to set environment variables manually, instead users need to execute just one simple command which defines the complete environment. The command that needs to be run from Bourne-type shell (e.g. bash) is:

. /reg/g/psdm/etc/ana_env.sh

For C-type shell (e.g. tcsh) the command should be:

source /reg/g/psdm/etc/ana_env.csh

The above command needs to be executed every time when a user logs in. For convenience it should be added to the user's shell login script. In case of bash shell here is the line of code that should be placed into ~/.bashrc file:

test -f /reg/g/psdm/etc/ana_env.sh && . /reg/g/psdm/etc/ana_env.sh

For tcsh the following code should be added to ~/.cshrc:

if ( -f /reg/g/psdm/etc/ana_env.csh ) source /reg/g/psdm/etc/ana_env.csh


Environment variables

This paragraph is informational and you can skip it if you are not interested in the internals of this setup. Various procedures, scripts, etc. depend on a small number of the environment variables which define location of various parts of the software and the system. Here are the variables that have to be present in the user's shell environment:

  • SIT_ENV – defines the type of the SIT environment, can be default for regular setup or ana for analysis environment
  • SIT_ROOT – defines the location of the top directory for the PSDM software, usually /reg/g/psdm.
  • SIT_RELDIR – directory with regular PSDM release
  • SIT_RELEASE – the name of the software release to use, something like "current" or "ana-1.1.5"
  • SIT_ARCH – the name of the hardware/software architecture to use like x86_64-rhel5-gcc41-opt
  • SIT_DATA – list of directories where the data files reside. These are not regular XTC data files, but rather some additional data that software needs. Usually it includes "<test-release>/data" and $SIT_RELDIR/$SIT_RELEASE/data (that is "data" directory in the releases) but can also be extended to include other locations.
  • SIT_REPOS – the locations of all "base" releases. Currently we only support single base release so this variable would look like $SIT_RELDIR/$SIT_RELEASE
  • LSF_PACKAGE_LIB – the location of the lsf package lib directory. Typical users will not set this. When at LCLS, it will be set to /afs/slac.stanford.edu/package/lsf/curr/lib/. Offsite users that have their own lsf installation they want to try with an ana release can set this appropriately.
  • SIT_NO_MODIFY_OMPI_ENV – typical users will not set this. Only offsite users that set OMPI environment variables may want to set this. see below.

Derived variables:

  • PATH – should be extended to include <test-release>/arch/$SIT_ARCH/bin and $SIT_RELDIR/$SIT_RELEASE/arch/$SIT_ARCH/bin
  • LD_LIBRARY_PATH – should be extended to include <test-release>/arch/$SIT_ARCH/lib and $SIT_RELDIR/$SIT_RELEASE/arch/$SIT_ARCH/lib, and $LSF_PACKAGE_LIB
  • PYTHONPATH – should be extended to include <test-release>/arch/$SIT_ARCH/python and $SIT_RELDIR/$SIT_RELEASE/arch/$SIT_ARCH/python

The <test-release> place holder is the absolute path to test release which is a release that is based on a base release (in $SIT_RELEASE). All these variables could be set manually, but there is a single convenience script which can be sourced from the user's shell which does all the tedious work.

MPI and LSF

The starting with ana 0.18.0, openmpi will try to load lsf components. sit_setup will set LSF_PACKAGE_LIB to the appropriate directory at SLAC. For offsite users, it will instead set three OMPI environment variables to instruct openmpi to not try to load the lsf components. These variables are

OMPI_MCA_ess = ^lsf
OMPI_MCA_ras  = ^lsf
OMPI_MCA_plm = ^lsf

Advanced users may want to set these variables themselves. If so, set

SIT_NO_MODIFY_OMPI_ENV=1

before setting up the ana environment.

Cern Root Users

At LCLS, the ana_env scripts make our cern root installation available to users. Advanced root users may want to run the thisroot.sh script to further setup the root environment. After running the ana_env.sh script, one would execute:

source thisroot.sh  # for bash, and thisroot.csh for c shell

however most root users should not need to do this. This script makes all the root utility programs available, and inserts root directories at the start of PATH, LD_LIBRARY_PATH and PYTHONPATH environment variables.

Setup scripts and their effect

Regular way to handle environment setup would be to use a special setup script which need to be sourced by user's shell. For convenience this can be included into a user's login script (.bashrc/.cshrc). The exact name of the script depends on the shell flavor and also on the type of the environment. For the data-management environment one needs to use:

(ba)sh: . /reg/g/psdm/etc/sit_env.sh
(t)csh: source /reg/g/psdm/etc/sit_env.csh

For analysis environment the commands are:

(ba)sh: . /reg/g/psdm/etc/ana_env.sh
(t)csh: source /reg/g/psdm/etc/ana_env.csh

By default script will use "current" as the release name (or "ana-current" for analysis environment), this can be changed if user's home directory contains file named SIT_RELEASE with the different release name in it.

Above scripts will define the alias sit_setup (sourcing the $SIT_ROOT/bin/sit_setup.{sh,csh}). This alias can be used to quickly change the environment variables to a new release or architecture. All path names to a release (test- and base-release) are using an absolute path. This means that changing a directory after running sit_setup will not change a release. When calling sit_setup zero or more parameters are accepted. The first step is to find out which release to use using the following order:

  1. if the first parameter to the script names a valid release (present in $SIT_RELDIR) the environment will be set to this release (no test-release)
  2. otherwise, if it is a path to a test release (a directory that contains .sit_release) it uses this path and the base release (read from .sit_release)
  3. otherwise, if the current working directory (cwd) contains .sit_release the cwd is used as test release and the environment is set as in the previous step
  4. otherwise if $HOME/SIT_RELEASE is present its content is used for the release name (must be a valid release present in $SIT_RELDIR)
  5. otherwise "current" or "ana-current" is used as the release name depending on environment type

The remaining parameters (that is all parameters except the first if the first parameter is a release name or test-release path) are used to build the SIT_ARCH value. SIT_ARCH is an ordered sequence of 4 values separated by dashes:

  • hardware architecture, such as i686, x86_64, etc.,
  • operating system brief name, such as rhel5, rhel6, rhel7, etc.,
  • compiler name and version, such as gcc41, gcc44, etc.,
  • debug/optimize/profile option, one of dbg, opt, prof.

The parameters to the sit_setup script can be used to specify one of the possible options for any of the four above items. If a parameter matches one of the possible options for the SIT_ARCH items, then that option is used in place of the default one. The simple example is:

% sit_setup          # sets SIT_ARCH to x86_64-rhel5-gcc41-opt

vs

% sit_setup dbg      # sets SIT_ARCH to x86_64-rhel5-gcc41-dbg

vs

% sit_setup dbg rhel5 # sets SIT_ARCH to x86_64-rhel5-gcc41-dbg

The SIT_ARCH options which are not specified on the command line are given their default values which depend on a particular platform/OS/release combination.

Additionally sit_setup accepts keyword options which provide explicit values for different parts of the SIT_ARCH variable. This is useful when the script is used for the setup of a completely new release, as in this case the script cannot always make a decision about such values as compiler name/version. The keyword options are:

  • -p processor – define processor type explicitly
  • -o os – define OS name explicitly
  • -c compiler – define compiler explicitly
  • -b build – define build type explicitly
  • -a name – define SIT_ARCH explicitly
  • -e env – overrides current value of SIT_ENV

The script needs to be executed again if a user wants to switch to a different release or to use different build option (debug vs optimized builds).

Obtaining Kerberos ticket

If you plan to work with code repository and use commands like svn, addpkg, etc. then for authorization you will need to obtain a Kerberos ticket. Ticket can be obtained at any time after you logged in by running command kinit which will prompt your for your regular password. Kerberos ticket is valid for 25 hours after which time one needs to re-run the command. Without ticket svn will print an error message "authorization failed" when it tries to access repository (some svn commands do not require repository access).

  • No labels