This page covers documentation about the conda release system for LCLS users.

Starting Conda

Users need to do the following

• use bash as your default shell, not tcsh or csh (conda doesn't support csh)
  • to switch to bash at LCLS (a separate computing environment than the rest of SLAC) email pcds-it-l at slac.stanford.edu
• Either
  
  • run the command: source /reg/g/psdm/etc/ana_env.sh as usual (explained in the psana python setup example)
  • run the command: source conda_setup
• Or, if you don't need to use the old RPM based release system
  • run the command: source /reg/g/psdm/bin/conda_setup

 Sourcing the conda_setup script does a few things

• modifies certain linux environment variables that can affect conda
• identifies the most recent conda environment built for LCLS data analysis
• activates that conda environment

At this point your linux prompt should change to something that starts with the name of the conda environment. For example,

(ana-1.0.7)

Python 3

The LCLS Data Analysis group will maintain conda environments with python 3. To activate the latest python 3 environment, pass the --py3 flag to conda_setup. That is run

source conda_setup --py3

Note - psana has not been ported to python 3. The python 3 environments do not include software to work with original LCLS experiment data files recorded by the data acquisition system (however you can work with translated hdf5 files).

Notes on Provided Conda Environments

• source conda_setup -h for full help on the script
• after sourcing conda_setup, you can execute the undo_conda command to restore your linux environment variables to what they were beforehand.
• see Conda Release Notes for release notes on the conda environments
• to switch to an older conda environment, for example to switch to ana-1.0.4, you can source conda_setup again, but now pass --env ana-1.0.4 to conda_setup. You can also, having sourced conda_setup once, use the standard conda command
  source activate ana-1.0.4
  to activate ana-1.0.4

Newer Package Versions

The conda environments will keep up with the latest versions for many standard python packages. Some notable version changes from the RPM based releases (as of December 2016):

• ipython, version 2.3 -> 5.1
• matplotlib, version 1.4.3 -> 1.5.1
• openmpi, version 1.8.3 -> 1.10.2
• mpi4py, version 1.3.1 -> 2.0.0
• numpy, version 1.9.2 -> 1.11.2 also our build is based on the intel MKL

New package versions means newer features, bug fixes, but it can also mean old interfaces are deprecated, and new bugs crop in. Some users may need to update their code to run with later versions of packages.

We do not use the latest version of packages that don't work with psana. 

Using conda_setup from a Script

When writing a script that will use the conda_setup command to create a conda environment, since conda_setup processes command line arguments, and your script may take command line arguments, a best practice is to do

source conda_setup ""

to prevent conda_setup from trying to read the scripts commands line arguments, details at conda_setup script issue.

Users may prefer to use the '–quiet' option from a script, that is

source conda_setup -q

to avoid conda_setup's messages - these messages can generate a lot of noise for a MPI script run on many cores.

While conda_setup should work from a script, it was designed for interactive use. This comment: bypass conda_setup in script shows how to setup the environment yourself, without using conda_setup.

Red Hat 5, 6, 7 Issues

Most users will run on rhel7 machines - anyone doing analysis on the interactive machines or submitting batch jobs to the usual queues will use rhel7 machines. There are some rhel6 and rhel5 machines in the experiment hutch control rooms. In particular users running from shared memory may have to run on a rhel6 or rhel5 host.

Continuum (which produced conda) suggests linux packages be built once in a way that runs on any linux operating system - rhel5, rhel6, rhel7, ubuntu, etc. In practice, for large packages like openmpi that the LCLS Data Analysis group must build local versions of, this can get very technical and lead to a brittle system. Similarly, conda-forge, a primary source of conda packages, decided to not support rhel5 for technical reasons.

The LCLS Data Analysis group is presently maintaining 3 separate conda installations, one for rhel5, rhel6, and rhel7. Locally built packages, like openmpi, hdf5, and psana, will be built natively on each platform. The conda_setup script will automatically detect rhel5 vs rhel6 vs rhel7 and activate an environment in the appropriate installation. Use conda_setup to get into a conda environment to make sure you use the appropriate packages for your host.

However since conda-forge does not support rhel5, certain packages will not function on rhel5 (for example, opencv, which we obtain from conda-forge). In general, our support for rhel5 is more limited with the conda releases than the RPM releases.

Moreover it is possible that we install a package that is limited to rhel7. As of Jan 2017, the only such package is tensorflow.

New Packages and Updates

It is much easier to add new packages to the LCLS conda environments than the RPM releases. Please contact us - pcds-ana-l@slac.stanford.edu for requests. Packages that are:

1. generally useful
2. come from standard conda channels like continuum defaults or conda-forge
3. do not introduce dependencies that interfere with psana

Will be straightforward to install.

Updating existing packages is straightforward - however we will always build new conda environments with a bumped up version number before changing an existing package.

Presently, packages that can only be installed via pip are not being added to the central installs, that is we require conda packaging, but see the User Conda Envionments section below.

User Conda Environments

Without Psana

A great way to experiment with a new package is to create your own conda environment and install the package there (this is similar to using virtualenv - which doesn't seem to work with our conda installations). For example, after doing

  source conda_setup

one could do, following conda – create an environment from the conda documentation (note, you need to be on a machine with outside interenet access like pslogin for the below command to work):

  conda create --name snowflake biopython

which installs biopythion from the the biopython conda package that is maintained by anaconda. Note this is different then using pip to install biopython. The nice thing about conda is that you can use pip to install packages in your conda environments. So for install, after creating your environment named snowflake, you could do

  source activate snowflake
  pip install PyAbel

With Psana

Many users will want to use various python packages with psana. The simplest way to do this is to clone an environment that has psana in it. The steps are, from a rhel7 machine with outside internet access (i.e, pslogin)

source conda_setup
conda create --name myenv --clone ana-1.0.8  # or whatever environment conda_setup put you in
source activate myenv
pip install PyAbel

Now switch to machine with access to the data (i.e, ssh psana) and do

source conda_setup
source activate myenv

To work with the data and psana at the same time. 

Notes:

• You cannot use your environment across rhel5, rhel6, and rhel7. You should only use it on rhel7. (Making three different environments named myenv-rhel6, myenv-rhel7 etc, won't work - they will all use the same psana-conda code
• The biggest issue, is that all 3GB of software in the LCLS conda environment will get copied into your home directory.

Supporting softlinks from user home directories to a central package repository is a feature under development at continuum.

Psana and Root

We're not sure if CERN Root will create problems for psana environments, however users who wish to use psana and root can try the following (root is in the non-standard channel NLeSC):

source conda_setup 
conda create --name myroot --clone ana-1.2.0  
source activate myroot 
conda install -c NLeSC root  

Updating Psana

To keep your envionment up to date with changes to psana, do

source activate myenv
conda update psana-conda

For example, if you cloned ana-1.0.8, but we are now at ana-1.0.9, the above would update your version of psana-conda. 

Offsite Installation

You should contact us pcds-ana-l@slac.stanford.edu before attempting offsite installation. We need to do development to automate this for users. Below is a recipe for manual installation to help users with offsite installation.

LCLS will maintain three channels in continuum:

lcls-rhel7
lcls-rhel6
lcls-rhel5

It is recommended to use the rhel7 channel to install on a rhel7 compatible version of linux.

On your offsite host computer, create a new environment for psana. Do the following:

conda create --name myenv  -c lcls-rhel7 psana-conda

this should install the latest version of psana-conda, and its dependencies. It should pick up the following dependencies from the lcls-rhel7 channel:

• hdf5
• openmpi
• mpi4py
• h5py
• tables

To use your own build of openmpi or hdf5/h5py, install those in myenv first (however note any package version requirements for running psana per the psana meta.yaml.

Before running psana, you will also need to set the environment variables

• SIT_PSDM_DATA
• SIT_ROOT
• SIT_DATA. 

The recommended way to do this is to have them set when your environment is activated, and unset when it is deactivated. Conda provides a mechanism to do this discussed here: saved-environment-variables. The complicated piece is that SIT_DATA must include the sub-directory 'data' to your conda environment, as well as where the experiment-db.dat file is. For instance, with a conda environment like ana-1.0.8, the files

  envs/ana-1.0.8/etc/conda/activate.d/env_vars.sh
  and
  envs/ana-1.0.8/etc/conda/deactivate.d/env_vars.sh 

read

  #!/bin/sh
  export SIT_DATA=/reg/g/psdm/sw/conda/inst/miniconda2-prod-rhel7/envs/ana-1.0.8/data:/reg/g/psdm/data
  export SIT_ARCH=x86_64-rhel7-gcc48-opt
  export SIT_ROOT=/reg/g/psdm

and

  #!/bin/sh
  unset SIT_DATA
  unset SIT_ARCH
  unset SIT_ROOT
  export SIT_DATA
  export SIT_ARCH
  export SIT_ROOT

respectively (SIT_ARCH is only used for building psana software). Note, we do not set SIT_PSDM_DATA since psana defaults to use /reg/d/psdm (where LCLS data is) if it is not defined.

Programmatic Version Information

Every ana conda environment that has the psana-conda package in it also has a package called

anarelinfo

that you can import from python to get

• The overall version of psana-conda in this conda environment
• The specific tag versions for the packages that comprise this version os psana-conda (this is more for psana developers than LCLS users)

GPU Work

LCLS has some GPU resources with some software setup for use. See table below

We are still developing infrastructure and configuration for these nodes, but presently, if one does

ssh psanagpu102
source conda_setup --dev --gpu

then you will be activating a python 2.7 conda environment for working with the GPU. It is mostly the same as the main environment with psana, but has these differences:

• includes the nvidia cudnn for deep learning frameworks like tensorflow
• adds paths to PATH, LD_LIBRARY_PATH, and CPATH so that you can work with the CUDA installation, and the nvidia cudnn
• for packages like tensorflow, that are compiled differently to work with the GPU, includes the GPU version of that package rather then the CPU version
  • presently, tensorflow is the only such package that is compiled differently for the GPU - that is all other packages in this environment are the same as the standard psana environment.
  • packages like theano can be dynamically configured to use the GPU, so it is the same package between this gpu and non gpu environment 

Using the cuDNN

Before using the  nvidia cudnn (by working with  tensorflow or keras in the gpu environment, or configuring  theano to use it), register with the NVIDIA Accelerated Computing Development program at this link:

 https://developer.nvidia.com/accelerated-computing-developer

Per the nvidia cuDNN license, we believe all users must register before using it, but don't worry, the nvidia emails (if you opt to receive them) are quite interesting!  

Shared Resource

Presently, the GPU's are only available through interactive nodes. There is no batch management of them to assign GPU resources to users. Be mindful that other users on a node like psanagpu102 may be using the GPU.

The main issue is that GPU memory can become a scarce resource.

Make use of the command

nvidia-smi

to see what other processes are on the gpu and how much memory they are using. Use 

top

to identify the names of other users and communicate with them, or us, to manage multi-use issues.

Limit GPU Card Use

If you are on a node with more than one GPU card, you can use cuda environment variables to restrict any CUDA based program, to only see a few of the GPU cards. For example, if there are two cards, they will be numbered 0 and 1 by CUDA. You could do 

export CUDA_VISIBLE_DEVICES=1

and any command you run will only see that one GPU. Likewise, to just start one process with a limited view, do

CUDA_VISIBLE_DEVICES=1 ipython

will start an interactive ipython session where tensorflow will only see device 1. Tensorflow will call the one device it sees device 0.

Configuration Subject to Change

At this point there are very few people using the GPU and the configuration of GPU support is subject to change. Presently the gpu conda environment is only built in the development rhel7 conda installation (thus the --dev switch for conda_setup above).