Page History

...

From an internet-accessible machine, download the latest Python 2.7.x source tarball (https://www.python.org/downloads/source/)
Create the Python installation directory structure in LCLS production: $PACKAGE_TOP/python/python<version> (e.g., $PACKAGE_TOP/python/python2.7.13),
which should look as follows:
Code Block
language bash
theme Midnight
$ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64 bin include lib man share src
Transfer the Python source tarball to $PACKAGE_TOP/python/python<version>/src:
Code Block
language bash
theme Midnight
$ ls /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src Python-2.7.13.tgz

Configure the build with the following options:

Anchor

	bootstrap_pip
	bootstrap_pip

Info
--prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 (To create all build directories under the current structure) --enable-shared (To allow building of shared libraries [.so files], required for packages such as PyQt) --with-ensurepip=upgrade (To bootstrap the latest pip installer for Python 2.7.x >= 2.7.9)

Code Block

language	bash
theme	Midnight

$ ./configure --prefix=$PACKAGE_TOP/python/python2.7.13/linux-x86_64 --enable-shared --with-ensurepip=upgrade
$ make
$ make install

Set the environment to use the newly-installed Python instance (example uses a script):
$ cat go_python2.7.13.bash
#!/bin/bash
# Sets the user environment to use Python2.7.13

if [ -d /afs/slac/g/lcls ]; then
export LCLS_ROOT=/afs/slac/g/lcls
else
export LCLS_ROOT=/usr/local/lcls
fi

export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1
export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64
export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}
export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${QT_HOME}/lib:${ORACLE_HOME}/lib:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}

$ source go_python2.7.13.bash
Check that the core installation includes pip and setuptools (required for building Python modules from source):
Code Block
language bash
theme Midnight
$ pip list
pip (9.0.1)
setuptools (28.8.0)

Python Package Installation

...

Core packages: packages for installing and building Python packages from sources or binaries (e.g. pip, setuptools, wheel) [$python_home/src/core-packages]
Site packages: packages obtained via the official Python Package Index (PyPI) repository ($python_home/src/site-packages)
External packages: packages obtained from non-PyPI sources, such as GitHub or SourceForge ($python_home/src/external-packages)
SLAC packages: packages created and/or maintained by SLAC entities ($python_home/src/slac-packages)

To install packages in production, run `pip install` and define the paths to the package source files with the --find-links option (note that pip will make a best effort to

satisfy dependencies, but this is by no means guaranteed. The --find-links argument can be specified more than once to look for files in multiple locations. Barring this,

the --no-deps argument can be used to ignore dependencies):

Code Block

language	bash
theme	Midnight

$# pip install --no-cache-dir --no-index --find-links=/path/to/module/source/files/dir [module|-r requirements.txt]
#
# Examples:
# Install using requirements file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages -r site-requirements.txt

# Install specific module (requires module tarball/wheel/egg file to be located in the directory specified by --find-links argument. If multiple packages
# match the name, the tarball/wheel/egg with the latest version number will be used.)
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages six

# Install module from specific source file
$ pip install --no-cache-dir --no-index --find-links=/usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages pydm-1.11.0.tar.gz

Each package subdirectory has its own requirements file, along with separate requirements files for packages with complex dependencies. The pip installer will avoid

installing duplicate packages if it finds tarballs <= the currently installed version.

Core packages

Core packages are required in order to install and build other Python packages from source. The pip and setuptools packages should be bootstrapped via the base

Python installation process (see above). The wheels package is required to install packages from wheels (.whl), which is necessary if the source tarball cannot be obtained.

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/core-packages/core-requirements.txt

...

pip

...


setuptools

...


wheel

Site packages

Site packages are available on the official Python Package Index (PyPI) repository. When possible, packages should be obtained from this repo for maintainability and

consistency. Python packages with complex dependencies have their own requirements files that are referenced by the top-level site-requirements.txt file. Note that

redundant packages are ignored in later requirements files if the dependency is already satisfied:

Flask

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/Flask-requirements.txt


Flask


Werkzeug


Jinja2


itsdangerous


click


MarkupSafe

IPython

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/ipython-requirements.txt


appdirs


backports.shutil-get-terminal-size


decorator


enum34


ipython


ipython-genutils


packaging


pathlib2


pexpect


pickleshare


prompt-toolkit


ptyprocess


pygments


pyparsing


scandir


setuptools


simplegeneric

six


traitlets


wcwidth

Jupyter

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/jupyter-requirements.txt


alabaster


appdirs


Babel


backports-abc


backports.shutil-get-terminal-size


backports.ssl-match-hostname


bleach


certifi


Cheetah


click


configparser


cx-Oracle


cycler


decorator


docutils


dpkt


edef


entrypoints


enum34


epics-batch-get


first


Flask


functools32


html5lib


imagesize


ipykernel


ipython


ipython-genutils


ipywidgets


itsdangerous


Jinja2


jsonschema


jupyter


jupyter-client


jupyter-console


jupyter-core


klystron


Markdown


MarkupSafe


matlog


matplotlib


meme


mistune


model


mpmath

mps


mpsbypass


nbconvert


nbformat


nose


notebook


numpy


orbit


packaging


pandas


pandocfilters


pathlib2


pexpect


physicselog


pickleshare

PIL

pip


pip-tools


plex


prompt-toolkit

psp


ptyprocess


pyca


pyepics


Pygments


pyparsing


pyqtgraph


PyQwt


python-dateutil


pytz


pyzmq


qtconsole


requests


roman


scandir


scikit-learn


scipy


setuptools


setuptools-scm


simplegeneric


singledispatch

six


snowballstemmer


Sphinx


subprocess32


subprocessca


sympy


terminado


testpath


tornado


traitlets


virtualenv


wcwidth


webencodings


Werkzeug


wheel


widgetsnbextension

pip-tools

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/pip-tools-requirements.txt


pip-tools


click


first


setuptools-scm

six

Scikit-learn

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/scikit-learn-requirements.txt


numpy


scipy


scikit-learn

Matplotlib

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/matplotlib-requirements.txt


cycler


functools32


matplotlib


numpy


pyparsing


python-dateutil


pytz

six


subprocess32

Top-level requirements*:
* - cx_Oracle >= 5.3.0 requires 64-bit Oracle 11.2 client or later. The proper ORACLE_HOME and related environment settings must be configured before
attempting to install the package via pip:

Code Block

language	bash
theme	Midnight

$ export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11source /usr/local/lcls/tools/oracle/oracleSetup-R11.2.0.2.0_x86_64/client_1
$ export PATH=${ORACLE_HOME}/bin:$PATH
$ export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:$LD_LIBRARY_PATH

$ cat

4.bash

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/site-packages/site-requirements.txt


# package-specific requirements



-r ./Flask-requirements.txt


-r ./ipython-requirements.txt


-r ./jupyter-requirements.txt


-r ./pip-tools-requirements.txt


-r ./scikit-learn-requirements.txt


-r ./matplotlib-requirements.txt



# global package requirements


Babel


Cheetah


Markdown


MarkupSafe


Pygments


alabaster


backports.ssl_match_hostname


backports_abc


certifi


cx_Oracle


decorator


docutils


dpkt


imagesize


mpmath


nose


pandas


plex


pyepics


pyqtgraph


pytz


requests


roman


singledispatch


snowballstemmer


sympy


tornado


virtualenv


wsgiref

External packages

External packages are ones that cannot be obtained from the PyPI repo. They can be obtained from project homepages (e.g., GitHub, SourceForge, etc).

...

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/external-packages/external-requirements.txt

...

PIL

...


PyQwt

...

sip

SLAC packages

SLAC packages are created and maintained by SLAC or affiliated contributors. Like external packages, these are available from various sources (GitHub, git, etc).

...

edlgenerator: https://github.com/slaclab/edlgenerator
mps_tools: https://github.com/slaclab/mps_tools
mpsbypass: https://github.com/slaclab/mpsbypass
psp: https://github.com/slaclab/psp
pyca: https://github.com/slaclab/pyca
slacepics: https://github.com/slaclab/slacepics
subprocessca: https://github.com/slaclab/subprocessca

Code Block

language	bash
theme	Midnight

$ cat /usr/local/lcls/package/python/python2.7.13/linux-x86_64/src/slac-packages/slac-requirements.txt

...


edlgenerator

...


mps_tools

...


mpsbypass

...

psp

...


pyca

...


slacepics

...


subprocessca

References:

Python 2.7: https://docs.python.org/2/
PyPI - The Python Package Index: https://pypi.python.org/pypi
pip - The Python Package Authority's preferred tool for installing packages: https://pip.pypa.io/en/stable/user_guide/#installing-packages
Jupyter: https://jupyter.org
PyQt: https://www.riverbankcomputing.com/software/pyqt/intro
Scikit-learn: http://scikit-learn.org/stable/index.html

...

Space shortcuts

Child pages

Versions Compared

Old Version 2

New Version 3

Key

Python Package Installation

Core packages

Site packages

External packages

SLAC packages