Developing an external python package typically means you will be doing something like
python setup.py develop somepackage
Without other options to setup.py, this will put a link (or links) from the python site-packages directory to the module directory (or directories) within somepackage.
This kind of development can be done by making your own conda environment. This is different than the central conda environments the LCLS Data analysis group maintains, or the test releases discussed on the Psana Developer Documentation page.
If you will not need psana in your environment, follow the instructions on creating an environment provided by continuum.
If you do need psana, then the steps will be
- create your own conda environment that includes psana
- HACK step: create bin subdir softlink in your local .conda
- activate this environment
- create a psana test release against this environment
- checkout the source for somepackage
- install it in develop mode in your environment
We'll go through these steps below.
1. Create Environment With Psana
The simplest way to create a environment with psana in it is to clone the ana-x.x.x environment that conda_setup puts you in. That is, after
source conda_setup
do
conda create --name myenv --clone ana-1.0.7
or whatever the environment is that conda put you in.
Notes
- This must be done from a rhel7 machine like pslogin that has outside internet access
- It takes a while
- It creates the hideen directory ~/.conda where your environments and package repository will be
- You will get 3GB + of packages
- There will be noisy warnings about conda.lock commands failing
- You can't use your environment on rhel5 or rhel6
Testing Your Environment
if you want to test your environment, enter the command
/reg/g/psdm/sw/conda/anarel-test/test_conda --bins --libs --imports --pkgs --pkglist all
2. HACK create soft link
test releases were not originally developed to be used against user environments, only centrally installed environments. So for the time being, you will need to mimic some of the directory structure in the central installs. The central installs have this directory structure:
conda_root
conda_root/bin
conda_root/envs
If you look in your .conda subdir, after creating your environment, you will see
~/.conda
~/.conda/envs
In this step, we will make a soft link to the bin subdir of the central install. If you look at the PATH environment variable with ana-1.0.7, this should be clear. You will do something like
cd ~/.conda
ln -s /reg/g/psdm/sw/conda/inst/miniconda2-prod-rhel7/bin bin
to make the soft link
~/.conda/bin --> /reg/g/psdm/sw/conda/inst/miniconda2-prod-rhel7/bin
Create Test Release Against Your Environment
In addition to creating test releases against conda environments managed by the LCLS Data Analaysis group, you can create a test release against your own environment. This will allow you to both install standard python packages in develop mode, and develop packages that are part of psana.
As disucces in Psana Developer Documentation, but now with your own conda environment active (i.e., myenv) rather then a centrally installed environment (ana-1.0.7), do
condarel --newrel --name mycondarel
This creates a test release directory mycondarel that runs against your myenv conda environment.
Checkout and Install
Now do the usual to checkout the package you want to develop, i.e,
- cd into your test release
- clone the package from github
- cd into the package
- run
python setup.py develop
Note, some packages may need additional options. For instance, to install h5py in this method, you must help it find the hdf5 installation. The hdf5 installation will be installed in your conda environment myenv. An easy way to specify this location for the h5py setup.py is with the CONDA_PREFIX environment variable that conda sets after it activates an environment. For instance, with h5py, one can do
python setup.py configure --mpi --hdf5=$CONDA_PREFIX
python setup.py develop
Conda_setup for Test Release and Develop
Now you activate your test release by doing
source conda_setup
from test release directory (mycondarel). If you did not do the HACK step correctly, this will fail.