Testing Python 2.7.13 64-bit in LCLS Production

EED Systems Group is working on upgrading the default Python instance in the LCLS production environment from Python 2.7.4 32-bit to Python 2.7.13 64-bit.

All Python applications in use in LCLS production must be tested and verified against the updated version before changing the default version of Python.

Testers can use the following environment settings to properly test their applications against Python 2.7.13 64-bit:

Configuring LCLS production environment for testing

Method 1

If you would like full control over your user environment settings, use the following instructions.

1. Add the new Python instance to your environment:

   export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64 

2. Add the new Python binaries to the PATH environment variable:

   export PATH=${QT_HOME}/bin:${ORACLE_HOME}/bin:${PYTHON_ROOT}/bin:${PATH}

3. Add the new Python libraries and their dependencies (EPICS 64-bit, Matlab, etc.) to the LD_LIBRARY_PATH environment variable. This is to ensure that the new 64-bit versions of Python modules built against those dependencies will work:

   export LD_LIBRARY_PATH=${LCLS_ROOT}/matlab/2016b/extern/lib:${EPICS_BASE_RELEASE}/lib/linux-x86_64:${EPICS_EXTENSIONS}/lib/linux-x86_64:${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${PYTHON_ROOT}/lib:${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH} 

4. Set the PYTHONPATH environment variable. This is to ensure that there are no false positives when testing applications, since the current PYTHONPATH may still have access to older versions of Python modules and override the new libraries:

   Note

   The ${EPICS_BASE_TOP}/base-cpp-R4-6-0/pvaPy/lib/python/2.7/linux-x86_64 directory contains EPICS V4 Python modules.
   The /usr/local/lcls/tools/python/toolbox directory contains SLAC-maintained Python modules.

   export PYTHONPATH=${EPICS_BASE_TOP}/base-cpp-R4-6-0/pvaPy/lib/python/2.7/linux-x86_64:${LCLS_ROOT}/tools/python/toolbox

5. [Optional] If your application(s) import the "cx_Oracle" module, add the following to your environment:

   export PATH=${ORACLE_HOME}/bin:${PATH}
   export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH} 

6. [Optional] If your application(s) import "PyQt", "PyQwt", or any other Python modules that depend on Qt4 64-bit, add the following to your environment:

   export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
   export PATH=${QT_HOME}/bin:${PATH}
   export LD_LIBRARY_PATH=${QT_HOME}/lib:${LD_LIBRARY_PATH} 

   
   If your application uses Qt5 64-bit, add the following to your environment (needs updated gcc and glibc):

   export LD_LIBRARY_PATH=${LCLS_ROOT}/package/gcc/gcc-4.9.4/lib:${LD_LIBRARY_PATH}
   export LD_LIBRARY_PATH=/usr/local/lcls/package/gcc/gcc-4.9.4/lib64:$LD_LIBRARY_PATH
    
   export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-5.9.1/linux-x86_64
   export PATH=${QT_HOME}/bin:${PATH}
   export LD_LIBRARY_PATH=${QT_HOME}/lib:${LD_LIBRARY_PATH}

Method 2

If you would like to use the full Python 2.7.13 64-bit environment and all of its dependencies, source the following setup script, which does all of the steps in Method 1, including the optional ones:


source /usr/local/lcls/tools/script/go_python2.7.13.bash

#!/bin/bash
# Sets the user environment to use Python2.7.13

if [ -d /afs/slac/g/lcls ]; then
    export LCLS_ROOT=/afs/slac/g/lcls
else
    export LCLS_ROOT=/usr/local/lcls
fi

# Set default Java to 64-bit
export JAVA_HOME=${LCLS_ROOT}/package/java/jdk1.8.0_144

if [ -z `echo "${PATH}" | grep "${JAVA_HOME}/bin"` ]; then
    export PATH=${JAVA_HOME}/bin:${PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${JAVA_HOME}/lib"` ]; then
    export LD_LIBRARY_PATH=${JAVA_HOME}/lib:${LD_LIBRARY_PATH}
fi

# Add Qt4 to 64-bit for PyQwt and PyQt4 compatibility
export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-4.8.6/linux-x86_64
if [ -z `echo "${PATH}" | grep "${QT_HOME}/bin"` ]; then
    export PATH=${QT_HOME}/bin:${PATH}
fi
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${QT_HOME}/lib"` ]; then
    export LD_LIBRARY_PATH=${QT_HOME}/lib:${LD_LIBRARY_PATH}
fi

# Add Qt5 64-bit for PyQwt and PyQt5 compatibility (requires updated gcc and glibc,
# sets default Qt to Qt5)
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${LCLS_ROOT}/package/gcc/gcc-4.9.4/lib"` ]; then
    export LD_LIBRARY_PATH=${LCLS_ROOT}/package/gcc/gcc-4.9.4/lib:${LD_LIBRARY_PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${LCLS_ROOT}/package/gcc/gcc-4.9.4/lib64"` ]; then
    export LD_LIBRARY_PATH=${LCLS_ROOT}/package/gcc/gcc-4.9.4/lib64:${LD_LIBRARY_PATH}
fi

export QT_HOME=${LCLS_ROOT}/package/Qt/Qt-5.9.1/linux-x86_64
if [ -z `echo "${PATH}" | grep "${QT_HOME}/bin"` ]; then
    export PATH=${QT_HOME}/bin:${PATH}
fi
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${QT_HOME}/lib"` ]; then
    export LD_LIBRARY_PATH=${QT_HOME}/lib:${LD_LIBRARY_PATH}
fi
# Set default Oracle client to 11.2.0.2.0 64-bit for cx_Oracle compatibility
export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.2.0.2.0_x86_64/client_1

if [ -z `echo "${PATH}" | grep "${ORACLE_HOME}/bin"` ]; then
    export PATH=${ORACLE_HOME}/bin:${PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${ORACLE_HOME}/lib"` ]; then
    export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH}
fi

# Set default Matlab to 2016b 64-bit for matlab_wrapper compatibility
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${LCLS_ROOT}/matlab/2016b/extern/lib"` ]; then
    export LD_LIBRARY_PATH=${LCLS_ROOT}/matlab/2016b/extern/lib:${LD_LIBRARY_PATH}
fi

# Set default EPICS to 64-bit for pyca and psp compatibility
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${EPICS_BASE_RELEASE}/lib/linux-x86_64"` ]; then
    export LD_LIBRARY_PATH=${EPICS_BASE_RELEASE}/lib/linux-x86_64:${LD_LIBRARY_PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${EPICS_EXTENSIONS}/lib/linux-x86_64"` ]; then
    export LD_LIBRARY_PATH=${EPICS_EXTENSIONS}/lib/linux-x86_64:${LD_LIBRARY_PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64"` ]; then
    export LD_LIBRARY_PATH=${EPICS_BASE_TOP}/base-cpp-R4-6-0/lib/linux-x86_64:${LD_LIBRARY_PATH}
fi

# Set default Python to 2.7.13 64-bit
export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.13/linux-x86_64

if [ -z `echo "${PATH}" | grep "${PYTHON_ROOT}/bin"` ]; then
    export PATH=${PYTHON_ROOT}/bin:${PATH}
fi

if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${PYTHON_ROOT}/lib"` ]; then
    export LD_LIBRARY_PATH=${PYTHON_ROOT}/lib:${LD_LIBRARY_PATH}
fi
if [ -z `echo "${LD_LIBRARY_PATH}" | grep "${PYTHON_ROOT}/build/Python2.7.13/Lib"` ]; then
    export LD_LIBRARY_PATH=${PYTHON_ROOT}/build/Python2.7.13/Lib:${LD_LIBRARY_PATH}
fi

# Update PYTHONPATH for EPICS V4 64-bit and SLAC-maintained module dependencies
export PYTHONPATH=${EPICS_BASE_TOP}/base-cpp-R4-6-0/pvaPy/lib/python/2.7/linux-x86_64:${LCLS_ROOT}/tools/python/toolbox

Workarounds

After testing, it may be discovered that certain applications may not run with Python 2.7.13 64-bit due to unresolved/conflicting dependencies (e.g., Python modules only available for 32-bit Python, Matlab2016b, Qt, etc.) or other reasons. If such dependencies cannot currently be resolved, Python developers may invoke the following environment settings in a wrapper script or similar to launch the application using the older instance of Python instead:

1. Add the previous Python instance (Python 2.7.4 32-bit) to your environment:

   export PYTHON_ROOT=${LCLS_ROOT}/package/python/python2.7.4

2. Add the previous Python binaries to your environment:

   export PATH=${PYTHON_ROOT}/bin:${PATH} 

3. Add the 32-bit Python libraries and their dependencies to your environment:

   export LD_LIBRARY_PATH=${PYTHON_ROOT}/lib:${LD_LIBRARY_PATH}
   export LD_LIBRARY_PATH=${PYTHON_ROOT}/lib/python2.7/lib-dynload:${LD_LIBRARY_PATH}

4. [Optional]  If your application(s) import the 32-bit "cx_Oracle" module, add the following to your environment:

   export ORACLE_HOME=${LCLS_ROOT}/package/oracle/product/11.1.0.6/client
   export PATH=${ORACLE_HOME}/bin:${PATH}
   export LD_LIBRARY_PATH=${ORACLE_HOME}/lib:${LD_LIBRARY_PATH}

5. [Optional] If your application(s) import "PyQt4", "PyQwt", or any other 32-bit Qt dependencies, add the following to your environment:

   export QT_HOME=${LCLS_ROOT}/package/Qt-4.8.5
   export PATH=${QT_HOME}/bin:${PATH}
   export LD_LIBRARY_PATH=${QT_HOME}/lib:${LD_LIBRARY_PATH}

6. [Optional] If your application(s) import "matlab_wrapper" or any other 32-bit Matlab 2012a dependent modules, add the following to your environment:

   export LD_LIBRARY_PATH=${LCLS_ROOT}/matlab/2012a/extern/lib:${LD_LIBRARY_PATH}