Python Deployment for LCLS Controls Environment

Overview

The following document describes the installation and deployment of Python and all required packages for the LCLS controls production environment.

The currently deployed versions are:

Python 2.7.4 32-bit (physics default)
Python 2.7.13 64-bit (softegr default)

Python deployment is primarily managed using pip, the Python Packaging Authority (PyPA) tool for installing Python packages.

Base Python Installation

The steps to install vanilla Python 2.7.x in LCLS production are as follows (examples use Python 2.7.13):

From an internet-accessible machine, download the latest Python 2.7.x source tarball (https://www.python.org/downloads/source/)
Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13),
which should look as follows:
```
$ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64
bin include lib man share src
```

Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:

$ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src
Python-2.7.13.tgz

Configure the build with the following options:

--prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure)
--enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt)
--with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)
```
$ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade
$ make
$ make install
```
Set the environment to use the newly-installed Python instance (example uses a script):
$ cat go_python2.7.13.bash
#!/bin/bash
# Sets the user environment to use Python2.7.13

if [ -d /afs/slac/g/lcls ]; then
export LCLS_ROOT=/afs/slac/g/lcls
else
export LCLS_ROOT=/usr/local/lcls
fi

export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1
export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}

$ source go_python2.7.13.bash
Check that the core installation includes pip and setuptools (required for building Python modules from source):
$ pip list
pip (9.0.1)
setuptools (28.8.0)

Python Package Installation

Python packages installed in LCLS controls production are obtained from various sources. When possible, it is preferable to build the packages from source in the

production environment to ensure compatibility.

There are four broad categories of Python packages deployed in production, depending on origin:

Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)

To install packages in production, run `pip install` and define the paths to the package source files with the --find-links option:

$ pip install --no-cache-dir --no-index --find-links=/path/to/module/source/files -r requirements.txt

Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid

installing duplicate packages if it finds tarballs <= the currently installed version.

Core packages

Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base

Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source cannot be obtained.

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/core-packages/core-requirements.txt
pip
setuptools
wheel

Site packages

Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and

consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file:

Flask
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/Flask-requirements.txt
Flask
Werkzeug
Jinja2
itsdangerous
click
MarkupSafe
IPython
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt
appdirs
backports.shutil-get-terminal-size
decorator
enum34
ipython
ipython-genutils
packaging
pathlib2
pexpect
pickleshare
prompt-toolkit
ptyprocess
pygments
pyparsing
scandir
setuptools
simplegeneric
six
traitlets
wcwidth
Jupyter
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt
alabaster
appdirs
Babel
backports-abc
backports.shutil-get-terminal-size
backports.ssl-match-hostname
bleach
certifi
Cheetah
click
configparser
cx-Oracle
cycler
decorator
docutils
dpkt
edef
entrypoints
enum34
epics-batch-get
first
Flask
functools32
html5lib
imagesize
ipykernel
ipython
ipython-genutils
ipywidgets
itsdangerous
Jinja2
jsonschema
jupyter
jupyter-client
jupyter-console
jupyter-core
klystron
Markdown
MarkupSafe
matlog
matplotlib
meme
mistune
model
mpmath
mps
mpsbypass
nbconvert
nbformat
nose
notebook
numpy
orbit
packaging
pandas
pandocfilters
pathlib2
pexpect
physicselog
pickleshare
PIL
pip
pip-tools
plex
prompt-toolkit
psp
ptyprocess
pyca
pyepics
Pygments
pyparsing
pyqtgraph
PyQwt
python-dateutil
pytz
pyzmq
qtconsole
requests
roman
scandir
scikit-learn
scipy
setuptools
setuptools-scm
simplegeneric
singledispatch
six
snowballstemmer
Sphinx
subprocess32
subprocessca
sympy
terminado
testpath
tornado
traitlets
virtualenv
wcwidth
webencodings
Werkzeug
wheel
widgetsnbextension
pip-tools
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt
pip-tools
click
first
setuptools-scm
six
Scikit-learn
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt
numpy
scipy
scikit-learn
Matplotlib
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt
cycler
functools32
matplotlib
numpy
pyparsing
python-dateutil
pytz
six
subprocess32
Top-level requirements*:

* - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before
attempting to install the package via pip:
```
$ export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1
$ export PATH=${ORACLE_HOME}/bin:$PATH
$ export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:$LD_LIBRARY_PATH 
```
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt
# package-specific requirements

-r ./Flask-requirements.txt
-r ./ipython-requirements.txt
-r ./jupyter-requirements.txt
-r ./pip-tools-requirements.txt
-r ./scikit-learn-requirements.txt
-r ./matplotlib-requirements.txt

# global package requirements
Babel
Cheetah
Markdown
MarkupSafe
Pygments
alabaster
backports.ssl_match_hostname
backports_abc
certifi
cx_Oracle
decorator
docutils
dpkt
imagesize
mpmath
nose
pandas
plex
pyepics
pyqtgraph
pytz
requests
roman
singledispatch
snowballstemmer
sympy
tornado
virtualenv
wsgiref

External packages

External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).

The external packages below are currently available from the following project sites:

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt
PIL
PyQwt
sip

SLAC packages

SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).

The SLAC-maintained packages below are available from the following project sites:

edlgenerator: https://github.com/slaclab/edlgenerator
mps_tools: https://github.com/slaclab/mps_tools
mpsbypass: https://github.com/slaclab/mpsbypass
psp: https://github.com/slaclab/psp
pyca: https://github.com/slaclab/pyca
slacepics: https://github.com/slaclab/slacepics
subprocessca: https://github.com/slaclab/subprocessca

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt
edlgenerator
mps_tools
mpsbypass
psp
pyca
slacepics
subprocessca

References:

Python 2.7: https://docs.python.org/2/
PyPI - The Python Package Index: https://pypi.python.org/pypi
pip - The Python Package Authority's preferred tool for installing packages: https://pip.pypa.io/en/stable/user_guide/#installing-packages
Jupyter: https://jupyter.org
PyQt: https://www.riverbankcomputing.com/software/pyqt/intro
Scikit-learn: http://scikit-learn.org/stable/index.html

Created by Patrick Pascual (ppascual@slac.stanford.edu)

Space shortcuts

Child pages

Overview

Base Python Installation

Python Package Installation

Core packages

Site packages