...
The following document describes the installation and deployment of Python and all required packages for the LCLS controls production environment.
The currently deployed versions are:
- Python 2.7.4 32-bit (physics default)
- Python 2.7.13 64-bit (softegr default)
Python deployment is primarily managed using pip, the Python Packaging Authority (PyPA) tool for installing Python packages.
Base Python Installation
...
The currently installed package list for Python 2.7.13 is here.
Warning |
---|
Please note that Python 2.7. |
...
x has reached end-of-life as of January, 2020 (see https://www.python.org/dev/peps/pep-0373/#id4). Python 2.7.13 deployment in the LCLS controls environment has been frozen and no new modules are to be installed as of March, 2021. Updates to existing modules that have 2.7 support are still allowed. |
Base Python Installation
The steps to install vanilla Python 2.7.x in LCLS production are as follows (examples use Python 2.7.13):
- From an internet-accessible machine, download the latest Python 2.7.13 source tarball (https://www.python.org/downloads/release/python-2713/)
Create the Python installation directory structure in LCLS production:
downloads/source/)Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13),
which should look as follows:Code Block language bash theme Midnight $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64 bin include lib man share src
Transfer the Python source tarball to $PACKAGE_TOP/python/python<version> /src:
(e.g., $PACKAGE_TOP/python/python2.7.13), which should look as follows:Code Block Code Block language bash theme Midnight $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src Python-2.7.13.tgz bin include lib man share src
Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:
Code Block language bash theme Midnight $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src Python-2.7.13.tgz
Configure the build with the following options:
Configure the build with the following options:Anchor bootstrap_pip bootstrap_pip Anchor bootstrap_pip bootstrap_pip Info --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure)
--enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt)
--with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)Code Block language bash theme Midnight $ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade $ make $ make install
Set Test the installation by setting the environment to use the newly-installed Python instance (example uses a script):
goCode Block language bash theme Midnight $ cat
init_python2.7.13.bash
#!/bin/bash
# Sets the user environment to use# Example of setting the user environment to use an instance of Python2.7.13
thenif [ -d /afs/slac/g/lcls ];
exportthen export LCLS_ROOT=/afs/slac/g/lcls
else
exportelse export LCLS_ROOT=/usr/local/lcls
fi
2.0_export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.
/client_14/linux-x86_64
4export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-
85.
69.
1/linux-x86_64
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}
go$ source
init_python2.7.13.bash
Check that the core installation includes pip and setuptools (required for building Python modules from source):
Code Block language bash theme Midnight $ pip list pip (9.0.1) setuptools (28.8.0)
...
Python packages installed in LCLS controls production are obtained from various sources. When possible, it is preferable to build the packages from source in the production environment to ensure compatibility.
...
- Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
- Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
- External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
- SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)
To install packages, run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to
satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this,
the --no-deps argument can be used to ignore dependencies):
...
:
...
From an internet-accessible machine, download the appropriate source tarballs (when available, otherwise omit the "--no-binary :all:" argument) from PyPI using pip:
Code Block language bash
theme Midnight #
...
Example: # Download source tarballs of all modules listed in requirements file to the destination directory $ pip
...
download --
...
dest=<download_directory> --no-
...
binary :all: -r
...
installed-
...
modules.txt #
...
Download source tarball of a specific module
...
to
...
the
...
destination
...
directory $
...
pip download --dest=<download_directory> --no-binary :all: six
- Copy the source files over to the appropriate directory in the Python 2.7.13 deployment tree (/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src) depending on package origin (see below).
- run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this, the --no-deps argument can be used to ignore dependencies):
Code Block | ||||
---|---|---|---|---|
| ||||
# directory specified by --find-links argument. If multiple packages # match the name, the tarball/wheel/egg with the latest version number will be used.) $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages six # Install module from specific source file $ pip install --no-cache-dir --no-index --find-links=/usrpath/localto/lclsmodule/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz |
Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid
installing duplicate packages if it finds tarballs <= the currently installed version.
...
Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base
Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat source/dir [module_source|-r requirements.txt] # # Examples: # Install using requirements file $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages -r site-requirements.txt # Install specific module (requires module tarball/wheel/egg file to be located in the directory specified by --find-links argument. If multiple packages # match the name, the tarball/wheel/egg with the latest version number will be used.) $ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/coresite-packages/core-requirements.txt pip setuptools wheel |
Site packages
Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and
consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that
redundant packages are ignored in later requirements files if the dependency is already satisfied:
Flask
...
six
# Install module from specific source file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz |
Info |
---|
Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid installing duplicate packages if it finds tarballs <= the currently installed version. |
Core packages
Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/ |
...
core-packages/ |
...
core-requirements.txt |
...
pip setuptools wheel |
Site packages
Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that redundant packages are ignored in later requirements files if the dependency is already satisfied:
Flask
Werkzeug
Jinja2
itsdangerous
click
MarkupSafeIPython
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt appdirs backports.shutil-get-terminal-size decorator enum34 ipython ipython-genutils packaging pathlib2 pexpect pickleshare prompt-toolkit ptyprocess pygments pyparsing scandir setuptools simplegeneric six traitlets wcwidth
Jupyter
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyterFlask-requirements.txt alabasterFlask appdirsWerkzeug Babel backports-abcJinja2 itsdangerous click MarkupSafe
IPython
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt appdirs backports.shutil-get-terminal-size decorator enum34 ipython ipython-genutils packaging pathlib2 pexpect pickleshare prompt-toolkit ptyprocess pygments pyparsing scandir setuptools simplegeneric six traitlets wcwidth
Jupyter
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt alabaster appdirs Babel backports-abc backports.shutil-get-terminal-size backports.ssl-match-hostname bleach certifi Cheetah click configparser cx-Oracle cycler decorator docutils dpkt entrypoints enum34 first Flask functools32 html5lib imagesize ipykernel ipython ipython-genutils ipywidgets itsdangerous Jinja2 jsonschema jupyter jupyter-client jupyter-console jupyter-core Markdown MarkupSafe matplotlib mistune mpmath nbconvert nbformat nose notebook numpy packaging pandas pandocfilters pathlib2 pexpectbackports.ssl-match-hostname bleach certifi Cheetah click configparser cx-Oracle cycler decorator docutils dpkt edef entrypoints enum34 epics-batch-get first Flask functools32 html5lib imagesize ipykernel ipython ipython-genutils ipywidgets itsdangerous Jinja2 jsonschema jupyter jupyter-client jupyter-console jupyter-core klystron Markdown MarkupSafe matlog matplotlib meme mistune model mpmath mps mpsbypass nbconvert nbformat nose notebook numpy orbit packaging pandas pandocfilters pathlib2 pexpect physicselog pickleshare PIL pip pip-tools plex prompt-toolkit psp ptyprocess pyca pyepics Pygments pyparsing pyqtgraph PyQwt python-dateutil pytz pyzmq qtconsole requests roman scandir scikit-learn scipy setuptools setuptools-scm simplegeneric singledispatch six snowballstemmer Sphinx subprocess32 subprocessca sympy terminado testpath tornado traitlets virtualenv wcwidth webencodings Werkzeug wheel widgetsnbextension
pip-tools
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt pip-tools click first setuptools-scm six
Scikit-learn
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt numpy scipy scikit-learn
Matplotlib
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt cycler functools32 matplotlib numpy pyparsing python-dateutil pytz six subprocess32
Top-level requirements*:
* - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before
attempting to install the package via pip:Code Block language bash theme Midnight $ source /usr/local/lcls/tools/oracle/oracleSetup-R11.2.0.4.bash
Code Block language bash theme Midnight $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt # package-specific requirements -r ./Flask-requirements.txt -r ./ipython-requirements.txt -r ./jupyter-requirements.txt -r ./pip-tools-requirements.txt -r ./scikit-learn-requirements.txt -r ./matplotlib-requirements.txt # global package requirements alembic Babel Cheetah Jinja2 Markdown MarkupSafe Pygments alabaster backports.ssl_match_hostname backports_abc certifi click cothread colorama cx_Oracle cycler decorator docutils dpkt first functools32 imagesize ipython ipython_genutils Mako mpmath nose numpy pandas pathlib2 pexpect pickleshare plex pydm pyepics == 3.2.7 pyqtgraph python-dateutil python-editor pytz PyYAML QtPy requests roman scipy scikit-learn simplegeneric singledispatch sip snowballstemmersix sympysnakefood tornadosnowballstemmer virtualenv wsgiref
External packages
External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).
The external packages below are currently available from the following project sites:
SQLAlchemy subprocess32 sympy tabulate tornado traitlets vcversioner virtualenv wsgiref -r ./jupyter-requirements.txt -r ./pip-tools-requirements.txt -r ./scikit-learn-requirements.txt -r ./matplotlib-requirements.txt
SLAC packages
SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).
The SLAC-maintained packages below are available from the following project sites:
- mps_tools: https://github.com/slaclab/mps_tools
- mpsbypass: https://github.com/slaclab/mpsbypass
- pyca: https://github.com/slaclab/pyca
- pydm: https://github.com/slaclab/pydm
- slacepics: https://github.com/slaclab/slacepics
- subprocessca: https://github.com/slaclab/subprocessca
- TimeChart: https://github.com/slaclab/timechart
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt
# deprecated: psp (included in pyca)
mps_tools
mpsbypass
pyca
slacepics
subprocessca
TimeChart |
External packages
External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).
The external packages below are currently available from the following project sites:
Code Block | ||||
---|---|---|---|---|
| ||||
$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt
PIL
PyQwt
|
PyQt5 and SIP
The PyQt5 module and its dependency module SIP (both maintained by Riverbank Computing) have special requirements and must be built using the following procedure:
Set the QT_HOME environment variable to the location of the Qt5 deployment in the LCLS Controls Environment:
Code Block language bash theme Midnight $ export QT_HOME=/usr/local/lcls/package/Qt/Qt-5.9.1/linux-x86_64
Ensure that the Python 2.7.13 deployment to build and install PyQt5 and SIP against is set as the default Python instance:
Code Block language bash theme Midnight $ export PATH=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/bin:$PATH $ export LD_LIBRARY_PATH=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/lib/python2.7/lib-dynload:/usr/local/lcls/package/python/python2.7.13/linux-x86_64/build/Python2.7.13/Lib:$LD_LIBRARY_PATH # test $ python --version Python 2.7.13
Extract and build the SIP module, then install it for the Python 2.7.13 instance:
- PIL: http://pythonware.com/products/pil/
- PyQwt: https://github.com/PyQwt
- SIP: https://www.riverbankcomputing.com/software/sip/download
Code Block language bash theme Midnight $
...
cd /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external
...
-packages $ tar -xzvf sip-4.19.6.tar.gz $ cd sip-4.19.6 $ python configure.py $ make $ make install
Set a proper gcc build chain:
Code Block language bash theme Midnight $ export PATH=/usr/local/lcls/package/gcc/gcc-4.9.4/bin:$PATH $ export LD_LIBRARY_PATH=/usr/local/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib:/usr/local/lcls/package/gcc/gcc-4.9.4/linux-x86_64/lib64:$LD_LIBRARY_PATH
Extract and build the PyQt5 module, then install it for the Python 2.7.13 intstance:
SLAC packages
SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).
The SLAC-maintained packages below are available from the following project sites:
- edlgenerator: https://github.com/slaclab/edlgenerator
- mps_tools: https://github.com/slaclab/mps_tools
- mpsbypass: https://github.com/slaclab/mpsbypass
- psp: https://github.com/slaclab/psp
- pyca: https://github.com/slaclab/pyca
- slacepics: https://github.com/slaclab/slacepics
- subprocessca: https://github.com/slaclab/subprocessca
Code Block language bash theme Midnight $
...
cd /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/
...
external-packages
...
$ tar -xzvf PyQt5_gpl-5.9.2.tar.gz $ cd PyQt5_gpl-5.9.2 $ python configure.py $ make $ make install
References:
- Python 2.7 (end-of-life as of January 2020): https://docs.python.org/2/
- PyPI - The Python Package Index: https://pypi.python.org/pypi
- pip - The Python Package Authority's preferred tool for installing packages: https://pip.pypa.io/en/stable/user_guide/#installing-packages
- Jupyter: https://jupyter.org
- PyQt: https://www.riverbankcomputing.com/software/pyqt/intro
- SIP: https://www.riverbankcomputing.com/software/sip/download
- Scikit-learn: http://scikit-learn.org/stable/index.html
...