BELOW DOESN'T WORK WELL - site-packages takes precedence on sys.path, need to use PYTHONPATH or uninstall orig - hassle

Developing an external python package typically means you will be doing something like

python setup.py develop somepackage

Without other options to setup.py, this will put a link (or links) from the python site-packages directory to the module directory (or directories) within somepackage.

This kind of development can be done by making your own conda environment. This is different than the central conda environments the LCLS Data analysis group maintains, or the test releases discussed on the Psana Developer Documentation page.

If you will not need psana in your environment, follow the instructions on creating an environment provided by continuum.

If you do need psana, then the steps will be

create your own conda environment that includes psana
HACK step: create bin subdir softlink in your local .conda
activate this environment
create a psana test release against this environment
checkout the source for somepackage
install it in develop mode in your environment

We'll go through these steps below.

1. Create Environment With Psana

The simplest way to create a environment with psana in it is to clone the ana-x.x.x environment that conda_setup puts you in. That is, after

source conda_setup

do

conda create --name myenv --clone ana-1.0.7

or whatever the environment is that conda put you in.

Notes

This must be done from a rhel7 machine like pslogin that has outside internet access
It takes a while
It creates the hidden directory ~/.conda where your environments and package repository will be
You will get 3GB + of packages
There will be noisy warnings about conda.lock commands failing
You can't use your environment on rhel5 or rhel6

Testing Your Environment

if you want to run the full test suite on your environment, enter the command

/reg/g/psdm/sw/conda/anarel-test/test_conda --bins --libs --imports --pkgs --pkglist all

2. HACK create soft link

test releases were not originally developed to be used against user environments, only centrally installed environments. So for the time being, you will need to mimic some of the directory structure in the central installs. The central installs have this directory structure:

conda_root
conda_root/bin
conda_root/envs

If you look in your .conda subdir, after creating your environment, you will see

~/.conda
~/.conda/envs

In this step, we will make a soft link to the bin subdir of the central install. Assuming you cloned a environment from the production rhel7 installation, you will do

cd ~/.conda
ln -s /reg/g/psdm/sw/conda/inst/miniconda2-prod-rhel7/bin bin

to make the soft link

~/.conda/bin --> /reg/g/psdm/sw/conda/inst/miniconda2-prod-rhel7/bin

Create Test Release Against Your Environment

In addition to creating test releases against conda environments managed by the LCLS Data Analaysis group, you can create a test release against your own environment (after doing hack step above). This will allow you to both install standard python packages in develop mode, and develop packages that are part of psana.

As disucces in Psana Developer Documentation, but now with your own conda environment active (i.e., myenv) rather then a centrally installed environment (ana-1.0.7), do

condarel --newrel --name mycondarel

This creates a test release directory mycondarel that runs against your myenv conda environment.

Checkout and Install

Now do the usual to checkout the package you want to develop, i.e,

cd into your test release
clone the package from github
cd into the package
run
python setup.py develop

Note, some packages may need additional options. For instance, to install h5py in this method, you must help it find the hdf5 installation. The hdf5 installation will be installed in your conda environment myenv. An easy way to specify this location for the h5py setup.py is with the CONDA_PREFIX environment variable that conda sets after it activates an environment. For instance, with h5py, one can do

python setup.py configure --mpi --hdf5=$CONDA_PREFIX
python setup.py develop

Inspect site-packages

One way to see if things worked, If you look in your myenv site-packages after installing in develop mode, you should see files like

ls -lrth $CONDA_PREFIX/lib/python2.7/site-packages

-rw-r--r-- 1 davidsch sc   46 Dec 13 22:00 h5py.egg-link
-rw-r--r-- 1 davidsch sc   53 Dec 13 22:02 scikit-beam.egg-link
-rw-r--r-- 1 davidsch sc   97 Dec 13 22:02 easy-install.pth

Conda_setup for Test Release and Develop

Now you activate your test release by doing