You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 6 Next »

Overview

The following document describes the installation and deployment of Python and all required packages for the LCLS controls production environment.

The currently deployed versions are:

  • Python 2.7.4 32-bit (physics default)
  • Python 2.7.13 64-bit (softegr default)

Python deployment is primarily managed using pip, the Python Packaging Authority (PyPA) tool for installing Python packages.

Base Python Installation

The steps to install vanilla Python 2.7.x in LCLS production are as follows (examples use Python 2.7.13):

  • From an internet-accessible machine, download the latest Python 2.7.x source tarball (https://www.python.org/downloads/source/)
  • Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13),
    which should look as follows:

    $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64
    bin include lib man share src
  • Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:

    $ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src
    Python-2.7.13.tgz
  • Configure the build with the following options:

    --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure)
    --enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt)
    --with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)

    $ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade
    $ make
    $ make install
  • Set the environment to use the newly-installed Python instance (example uses a script):

    $ cat go_python2.7.13.bash
    #!/bin/bash
    # Sets the user environment to use Python2.7.13

    if [ -d /afs/slac/g/lcls ]; then
        export LCLS_ROOT=/afs/slac/g/lcls
    else
        export LCLS_ROOT=/usr/local/lcls
    fi

    export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1
    export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
    export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
    export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
    export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}

    $ source go_python2.7.13.bash


  • Check that the core installation includes pip and setuptools (required for building Python modules from source):

    $ pip list
    pip (9.0.1)
    setuptools (28.8.0)

Python Package Installation

Python packages installed in LCLS controls production are obtained from various sources. When possible, it is preferable to build the packages from source in the

production environment to ensure compatibility.

There are four broad categories of Python packages deployed in production, depending on origin:

  • Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
  • Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
  • External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
  • SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)

To install packages, run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to

satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this,

the --no-deps argument can be used to ignore dependencies):

# pip install --no-cache-dir --no-index --find-links=/path/to/module/source/dir [module|-r requirements.txt]
#
# Examples:
# Install using requirements file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages -r site-requirements.txt

# Install specific module (requires module tarball/wheel/egg file to be located in the directory specified by --find-links argument. If multiple packages
# match the name, the tarball/wheel/egg with the latest version number will be used.)
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages six

# Install module from specific source file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz

Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid

installing duplicate packages if it finds tarballs <= the currently installed version.


Core packages

Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base

Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/core-packages/core-requirements.txt
pip
setuptools
wheel


Site packages

Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and

consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that

redundant packages are ignored in later requirements files if the dependency is already satisfied:

  • Flask

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/Flask-requirements.txt
    Flask
    Werkzeug
    Jinja2
    itsdangerous
    click
    MarkupSafe
  • IPython

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt
    appdirs
    backports.shutil-get-terminal-size
    decorator
    enum34
    ipython
    ipython-genutils
    packaging
    pathlib2
    pexpect
    pickleshare
    prompt-toolkit
    ptyprocess
    pygments
    pyparsing
    scandir
    setuptools
    simplegeneric
    six
    traitlets
    wcwidth
  • Jupyter

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt
    alabaster
    appdirs
    Babel
    backports-abc
    backports.shutil-get-terminal-size
    backports.ssl-match-hostname
    bleach
    certifi
    Cheetah
    click
    configparser
    cx-Oracle
    cycler
    decorator
    docutils
    dpkt
    entrypoints
    enum34
    first
    Flask
    functools32
    html5lib
    imagesize
    ipykernel
    ipython
    ipython-genutils
    ipywidgets
    itsdangerous
    Jinja2
    jsonschema
    jupyter
    jupyter-client
    jupyter-console
    jupyter-core
    Markdown
    MarkupSafe
    matplotlib
    mistune
    mpmath
    nbconvert
    nbformat
    nose
    notebook
    numpy
    packaging
    pandas
    pandocfilters
    pathlib2
    pexpect
    pickleshare
    PIL
    pip
    pip-tools
    plex
    prompt-toolkit
    ptyprocess
    Pygments
    pyparsing
    pyqtgraph
    PyQwt
    python-dateutil
    pytz
    pyzmq
    qtconsole
    requests
    roman
    scandir
    scikit-learn
    scipy
    setuptools
    setuptools-scm
    simplegeneric
    singledispatch
    six
    snowballstemmer
    Sphinx
    subprocess32
    sympy
    terminado
    testpath
    tornado
    traitlets
    virtualenv
    wcwidth
    webencodings
    Werkzeug
    wheel
    widgetsnbextension
  • pip-tools

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt
    pip-tools
    click
    first
    setuptools-scm
    six
  • Scikit-learn

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt
    numpy
    scipy
    scikit-learn
  • Matplotlib

    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt
    cycler
    functools32
    matplotlib
    numpy
    pyparsing
    python-dateutil
    pytz
    six
    subprocess32
  • Top-level requirements*:
    * - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before
    attempting to install the package via pip:

    $ source /usr/local/lcls/tools/oracle/oracleSetup-R11.2.0.4.bash
    $ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt
    # package-specific requirements
    
    -r ./Flask-requirements.txt
    -r ./ipython-requirements.txt
    
    # global package requirements
    alembic
    Babel
    Cheetah
    Jinja2
    Markdown
    MarkupSafe
    Pygments
    alabaster
    backports.ssl_match_hostname
    backports_abc
    certifi
    click
    cothread
    colorama
    cx_Oracle
    cycler
    decorator
    docutils
    dpkt
    first
    functools32
    imagesize
    ipython
    ipython_genutils
    Mako
    mpmath
    nose
    numpy
    pandas
    pathlib2
    pexpect
    pickleshare
    plex
    pydm
    pyepics == 3.2.7
    pyqtgraph
    python-dateutil
    python-editor
    pytz
    PyYAML
    QtPy
    requests
    roman
    scipy
    scikit-learn
    simplegeneric
    singledispatch
    sip
    six
    snakefood
    snowballstemmer
    SQLAlchemy
    subprocess32
    sympy
    tabulate
    tornado
    traitlets
    vcversioner
    virtualenv
    wsgiref
    
    -r ./jupyter-requirements.txt
    -r ./pip-tools-requirements.txt
    -r ./scikit-learn-requirements.txt
    -r ./matplotlib-requirements.txt


SLAC packages

SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).

The SLAC-maintained packages below are available from the following project sites:

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt
# deprecated: psp (included in pyca)
mps_tools
mpsbypass
pyca
slacepics
subprocessca
TimeChart 


External packages

External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).

The external packages below are currently available from the following project sites:

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt
PIL
PyQwt
sip


PyQt5

Please see instructions for installing PyQt5 here.


References:




Created by Patrick Pascual (ppascual@slac.stanford.edu)



  • No labels