Overview

The following document describes the installation and deployment of Python and all required packages for the LCLS controls environment.

The currently deployed versions are:

  • Python 2.7.4 32-bit (physics default)
  • Python 2.7.13 64-bit (softegr default)

The currently installed package list for Python 2.7.13 is here.

Please note that Python 2.7.x has reached end-of-life as of January, 2020 (see https://www.python.org/dev/peps/pep-0373/#id4). Python 2.7.13 deployment in the LCLS controls environment has been frozen and no new modules are to be installed as of March, 2021. Updates to existing modules that have 2.7 support are still allowed.

Base Python Installation

The steps to install vanilla Python 2.7.x in LCLS production are as follows (examples use Python 2.7.13):

  • From an internet-accessible machine, download the latest Python 2.7.13 source tarball (https://www.python.org/downloads/release/python-2713/)

  • Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13), which should look as follows:

    $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64
bin include lib man share src

  • Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:

    $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src
Python-2.7.13.tgz

  • Configure the build with the following options:

    --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure)
    --enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt)
    --with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)

    $ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade
$ make
$ make install

  • Test the installation by setting the environment to use the newly-installed Python instance (example uses a script):

    $ cat init_python2.7.13.bash
#!/bin/bash
# Example of setting the user environment to use an instance of Python2.7.13

if [ -d /afs/slac/g/lcls ]; then
    export LCLS_ROOT=/afs/slac/g/lcls
else
    export LCLS_ROOT=/usr/local/lcls
fi

export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.4/linux-x86_64
export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-5.9.1/linux-x86_64
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}

$ source init_python2.7.13.bash


  • Check that the core installation includes pip and setuptools (required for building Python modules from source):

    $ pip list
pip (9.0.1)
setuptools (28.8.0)

Python Package Installation

Python packages installed in LCLS controls production are obtained from various sources. When possible, it is preferable to build the packages from source in the production environment to ensure compatibility.

There are four broad categories of Python packages deployed in production, depending on origin:

  • Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
  • Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
  • External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
  • SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)

To install packages:

  • From an internet-accessible machine, download the appropriate source tarballs (when available, otherwise omit the "--no-binary :all:" argument) from PyPI using pip:

    # Example:
# Download source tarballs of all modules listed in requirements file to the destination directory
$ pip download --dest=<download_directory> --no-binary :all: -r installed-modules.txt

# Download source tarball of a specific module to the destination directory
$ pip download --dest=<download_directory> --no-binary :all: six
  • Copy the source files over to the appropriate directory in the Python 2.7.13 deployment tree (/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src) depending on package origin (see below).
  • run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this, the --no-deps argument can be used to ignore dependencies):
# pip install --no-cache-dir --no-index --find-links=/path/to/module/source/dir [module_source|-r requirements.txt]
#
# Examples:
# Install using requirements file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages -r site-requirements.txt

# Install specific module (requires module tarball/wheel/egg file to be located in the directory specified by --find-links argument. If multiple packages
# match the name, the tarball/wheel/egg with the latest version number will be used.)
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages six

# Install module from specific source file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz

Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid installing duplicate packages if it finds tarballs <= the currently installed version.


Core packages

Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/core-packages/core-requirements.txt
pip
setuptools
wheel


Site packages

Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that redundant packages are ignored in later requirements files if the dependency is already satisfied:

  • Flask

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/Flask-requirements.txt
Flask
Werkzeug
Jinja2
itsdangerous
click
MarkupSafe

  • IPython

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt
appdirs
backports.shutil-get-terminal-size
decorator
enum34
ipython
ipython-genutils
packaging
pathlib2
pexpect
pickleshare
prompt-toolkit
ptyprocess
pygments
pyparsing
scandir
setuptools
simplegeneric
six
traitlets
wcwidth

  • Jupyter

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt
alabaster
appdirs
Babel
backports-abc
backports.shutil-get-terminal-size
backports.ssl-match-hostname
bleach
certifi
Cheetah
click
configparser
cx-Oracle
cycler
decorator
docutils
dpkt
entrypoints
enum34
first
Flask
functools32
html5lib
imagesize
ipykernel
ipython
ipython-genutils
ipywidgets
itsdangerous
Jinja2
jsonschema
jupyter
jupyter-client
jupyter-console
jupyter-core
Markdown
MarkupSafe
matplotlib
mistune
mpmath
nbconvert
nbformat
nose
notebook
numpy
packaging
pandas
pandocfilters
pathlib2
pexpect
pickleshare
PIL
pip
pip-tools
plex
prompt-toolkit
ptyprocess
Pygments
pyparsing
pyqtgraph
PyQwt
python-dateutil
pytz
pyzmq
qtconsole
requests
roman
scandir
scikit-learn
scipy
setuptools
setuptools-scm
simplegeneric
singledispatch
six
snowballstemmer
Sphinx
subprocess32
sympy
terminado
testpath
tornado
traitlets
virtualenv
wcwidth
webencodings
Werkzeug
wheel
widgetsnbextension

  • pip-tools

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt
pip-tools
click
first
setuptools-scm
six

  • Scikit-learn

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt
numpy
scipy
scikit-learn

  • Matplotlib

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt
cycler
functools32
matplotlib
numpy
pyparsing
python-dateutil
pytz
six
subprocess32

  • Top-level requirements*:
    * - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before attempting to install the package via pip:

    $ source /usr/local/lcls/tools/oracle/oracleSetup-R11.2.0.4.bash
    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt
# package-specific requirements

-r ./Flask-requirements.txt
-r ./ipython-requirements.txt

# global package requirements
alembic
Babel
Cheetah
Jinja2
Markdown
MarkupSafe
Pygments
alabaster
backports.ssl_match_hostname
backports_abc
certifi
click
cothread
colorama
cx_Oracle
cycler
decorator
docutils
dpkt
first
functools32
imagesize
ipython
ipython_genutils
Mako
mpmath
nose
numpy
pandas
pathlib2
pexpect
pickleshare
plex
pydm
pyepics == 3.2.7
pyqtgraph
python-dateutil
python-editor
pytz
PyYAML
QtPy
requests
roman
scipy
scikit-learn
simplegeneric
singledispatch
sip
six
snakefood
snowballstemmer
SQLAlchemy
subprocess32
sympy
tabulate
tornado
traitlets
vcversioner
virtualenv
wsgiref

-r ./jupyter-requirements.txt
-r ./pip-tools-requirements.txt
-r ./scikit-learn-requirements.txt
-r ./matplotlib-requirements.txt

SLAC packages

SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).

The SLAC-maintained packages below are available from the following project sites:

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt
# deprecated: psp (included in pyca)
mps_tools
mpsbypass
pyca
slacepics
subprocessca
TimeChart


External packages

External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).

The external packages below are currently available from the following project sites:

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt
PIL
PyQwt


PyQt5 and SIP

The PyQt5 module and its dependency module SIP (both maintained by Riverbank Computing) have special requirements and must be built using the following procedure:

  • Set the QT_HOME environment variable to the location of the Qt5 deployment in the LCLS Controls Environment:

    $ export QT_HOME=/usr/local/lcls/package/Qt/Qt-5.9.1/linux-x86_64

  • Ensure that the Python 2.7.13 deployment to build and install PyQt5 and SIP against is set as the default Python instance:

    $ export PATH=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/bin:$PATH
$ export LD_LIBRARY_PATH=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib/python2.7/lib-dynload:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/build/Python2.7.13/Lib:$LD_LIBRARY_PATH

# test
$ python --version
Python 2.7.13

  • Extract and build the SIP module, then install it for the Python 2.7.13 instance:

    $ cd /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages
$ tar -xzvf sip-4.19.6.tar.gz
$ cd sip-4.19.6
$ python configure.py
$ make
$ make install

  • Set a proper gcc build chain:

    $ export PATH=/usr/local/lcls/package/gcc/gcc-4.9.4/bin:$PATH
$ export LD_LIBRARY_PATH=/usr/local/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib:/usr/local/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib64:$LD_LIBRARY_PATH

  • Extract and build the PyQt5 module, then install it for the Python 2.7.13 intstance:

    $ cd /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages
$ tar -xzvf PyQt5_gpl-5.9.2.tar.gz
$ cd PyQt5_gpl-5.9.2
$ python configure.py
$ make
$ make install


References:


Created by Patrick Pascual (ppascual@slac.stanford.edu)



  • No labels