Overview
The following document describes the installation and deployment of Python and all required packages for the LCLS controls production environment.
The currently deployed versions are:
- Python 2.7.4 32-bit (physics default)
- Python 2.7.13 64-bit (softegr default)
Python deployment is primarily managed using pip, the Python Packaging Authority (PyPA) tool for installing Python packages.
Base Python Installation
The steps to install vanilla Python 2.7.x in LCLS production are as follows (examples use Python 2.7.13):
- From an internet-accessible machine, download the latest Python 2.7.x source tarball (https://www.python.org/downloads/source/)
Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13),
which should look as follows:Code Block language bash theme Midnight $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64 bin include lib man share src
Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:
Code Block language bash theme Midnight $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src Python-2.7.13.tgz
Configure the build with the following options:
Anchor bootstrap_pip bootstrap_pip Info --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure)
--enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt)
--with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)Code Block language bash theme Midnight $ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade $ make $ make install
Set the environment to use the newly-installed Python instance (example uses a script):
$ cat go_python2.7.13.bash
#!/bin/bash
# Sets the user environment to use Python2.7.13
if [ -d /afs/slac/g/lcls ]; then
export LCLS_ROOT=/afs/slac/g/lcls
else
export LCLS_ROOT=/usr/local/lcls
fi
export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1
export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}
$ source go_python2.7.13.bashCheck that the core installation includes pip and setuptools (required for building Python modules from source):
Code Block language bash theme Midnight $ pip list pip (9.0.1) setuptools (28.8.0)
Python Package Installation
Python packages installed in LCLS controls production are obtained from various sources. When possible, it is preferable to build the packages from source in the
production environment to ensure compatibility.
There are four broad categories of Python packages deployed in production, depending on origin:
- Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
- Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
- External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
- SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)
To install packages, run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to
satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this,
the --no-deps argument can be used to ignore dependencies):
Code Block | ||||
---|---|---|---|---|
| ||||
# pip install --no-cache-dir --no-index --find-links=/path/to/module/source/dir [module|-r requirements.txt] # # Examples: # Install using requirements file $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages -r site-requirements.txt # Install specific module (requires module tarball/wheel/egg file to be located in the directory specified by --find-links argument. If multiple packages # match the name, the tarball/wheel/egg with the latest version number will be used.) $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages six # Install module from specific source file $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz |
Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid
installing duplicate packages if it finds tarballs <= the currently installed version.
Core packages
Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base
Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/core-packages/core-requirements.txt pip setuptools wheel |
Site packages
Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and
consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that
redundant packages are ignored in later requirements files if the dependency is already satisfied:
Flask
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/Flask-requirements.txt Flask Werkzeug Jinja2 itsdangerous click MarkupSafe
IPython
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt appdirs backports.shutil-get-terminal-size decorator enum34 ipython ipython-genutils packaging pathlib2 pexpect pickleshare prompt-toolkit ptyprocess pygments pyparsing scandir setuptools simplegeneric six traitlets wcwidth
Jupyter
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt alabaster appdirs Babel backports-abc backports.shutil-get-terminal-size backports.ssl-match-hostname bleach certifi Cheetah click configparser cx-Oracle cycler decorator docutils dpkt edef entrypoints enum34 epics-batch-get first Flask functools32 html5lib imagesize ipykernel ipython ipython-genutils ipywidgets itsdangerous Jinja2 jsonschema jupyter jupyter-client jupyter-console jupyter-core klystron Markdown MarkupSafe matlog matplotlib meme mistune model mpmath mps mpsbypass nbconvert nbformat nose notebook numpy orbit packaging pandas pandocfilters pathlib2 pexpect physicselog pickleshare PIL pip pip-tools plex prompt-toolkit psp ptyprocess pyca pyepics Pygments pyparsing pyqtgraph PyQwt python-dateutil pytz pyzmq qtconsole requests roman scandir scikit-learn scipy setuptools setuptools-scm simplegeneric singledispatch six snowballstemmer Sphinx subprocess32 subprocessca sympy terminado testpath tornado traitlets virtualenv wcwidth webencodings Werkzeug wheel widgetsnbextension
pip-tools
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt pip-tools click first setuptools-scm six
- Scikit-learn
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt numpy scipy scikit-learn
Matplotlib
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt cycler functools32 matplotlib numpy pyparsing python-dateutil pytz six subprocess32
Top-level requirements*:
* - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before
attempting to install the package via pip:Code Block language bash theme Midnight $ source /usr/local/lcls/tools/oracle/oracleSetup-R11.2.0.4.bash
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt # package-specific requirements -r ./Flask-requirements.txt -r ./ipython-requirements.txt -r ./jupyter-requirements.txt -r ./pip-tools-requirements.txt -r ./scikit-learn-requirements.txt -r ./matplotlib-requirements.txt # global package requirements Babel Cheetah Markdown MarkupSafe Pygments alabaster backports.ssl_match_hostname backports_abc certifi cx_Oracle decorator docutils dpkt imagesize mpmath nose pandas plex pyepics pyqtgraph pytz requests roman singledispatch snowballstemmer sympy tornado virtualenv wsgiref
External packages
External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).
The external packages below are currently available from the following project sites:
- PIL: http://pythonware.com/products/pil/
- PyQwt: https://github.com/PyQwt
- SIP: https://www.riverbankcomputing.com/software/sip/download
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt PIL PyQwt sip |
SLAC packages
SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).
The SLAC-maintained packages below are available from the following project sites:
- edlgenerator: https://github.com/slaclab/edlgenerator
- mps_tools: https://github.com/slaclab/mps_tools
- mpsbypass: https://github.com/slaclab/mpsbypass
- psp: https://github.com/slaclab/psp
- pyca: https://github.com/slaclab/pyca
- slacepics: https://github.com/slaclab/slacepics
- subprocessca: https://github.com/slaclab/subprocessca
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt edlgenerator mps_tools mpsbypass psp pyca slacepics subprocessca |
References:
- Python 2.7: https://docs.python.org/2/
- PyPI - The Python Package Index: https://pypi.python.org/pypi
- pip - The Python Package Authority's preferred tool for installing packages: https://pip.pypa.io/en/stable/user_guide/#installing-packages
- Jupyter: https://jupyter.org
- PyQt: https://www.riverbankcomputing.com/software/pyqt/intro
- Scikit-learn: http://scikit-learn.org/stable/index.html
Created by Patrick Pascual (ppascual@slac.stanford.edu)