Overview

The Package Manager is a  system of scripts, configuration files, python files, and tarballs, version controlled via subversion, that facilitate building general purpose linux packages for different architectures.  It is intended to replace the current collection of hand built and maintained packages in /reg/g/pcds/package, /afs/slac/g/pcds/package, and /afs/slac/g/lcls/package.   Currently, only host builds are supported, but soon we will likely be adding cross compile targets, in particular linuxRT cross compiles.

Setting PSPKG_ROOT in your environment.

Before using the Package Manager, you need to set the environment variable PSPKG_ROOT.

There are currently two actively supported paths for PSPKG_ROOT.

On the NEH NFS filesystem, set PSPKG_ROOT to /reg/g/pcds/pkg_mgr.

For servers that don't have access to the NEH NFS filesystem, but can access the SLAC AFS filesystem, set PSPKG_ROOT to /afs/slac/g/pcds/pkg_mgr.

What's under $PSPKG_ROOT

• README_ps_make - Short readme file w/ the basics on how to use the package manager
• ps_make - Short bash script that launches the package manager, tee'ing the build output to build.log
• tarballs - Directory containing various *.gz, *.bz2, *.xz, etc tarballs.  Generally these are the download filenames, but occasionally some renaming is needed to PACKAGE-VERS.EXTENSION
• source/scripts - Directory containing the package manager python scripts
• source/pkg_info - Directory containing a dependency config file and a package specific python script for each package.  The python script inherits from standard unix_package or python_package classes, adding any custom options needed to configure, build, and/or install it.
• source/releases - Directory containing release configuration files.   A release is a collection of specific package versions.
• release - Directory containing all packages and/or releases which have been built.  Paths follow the pattern $PSPKG_ROOT/release/PACKAGE/VERSION/TARGET_ARCH

All of the above except release are under version control using subversion.   All PCDS and LCLS developers should have r/w access to the repo via the AFS filesystem.

The repository path for the package manager is file:///afs/slac/g/pcds/vol2/svn/pcds/package/trunk/pkg_mgr

Building Packages

Packages and releases are built with the ps_make script.  To build one particular package for the current architecture, run:

${PSPKG_ROOT}/ps_make -p PACKAGE/VERSION

where PACKAGE is the name of the page and VERSION is the particular version.  For example to build version 2.2 of the tornado package, run:

${PSPKG_ROOT}/ps_make -p tornado/2.2

 To build the xpp-1.0 release, do:

${PSPKG_ROOT}/ps_make -r RELEASE

where RELEASE is the name of the release specification file for xpp-1.0. E.g.:

${PSPKG_ROOT}/ps_make -r xpp-1.0

Packages will be built for the current architecture (e.g. x86_64-rhel5-gcc41-opt if you are building on psdev01.slac.stanford.edu). Only those packages not already built will be built. When a package is successfully built, a file is created in the package directory, e.g.:

${PSPKG_ROOT}/release/xraylib/2.15.0-python2.7/x86_64-rhel5-gcc41-opt/.xraylib_is_installed

Under ${PSPKG_ROOT}/source/pkg_info there are two files for each package. E.g. for scipy the files are:

pkg_info/scipy.py
pkg_info/scipy.dep

The first file contains a python class containing code to build the package. The second file contains dependency information for the package. E.g. scipy.dep contains:

scipy/0.11.0: python/2.7.3, atlas/3.8.3, numpy/1.6.2

When ps_make is run, it first checks the dependencies for every package listed in the release. If a dependency is found that is not in the release specification file, it will be added to the list of packages to build (if not already built). Thus when building xpp-1.0, you will see the output like:

Loading release list from /reg/g/pcds/pkg_mgr/source/scripts/release_xpp-1.0.rel...
Loading tcl-tk 8.5.12 (required by python 2.7.3)
Loading python 2.7.3 (required by boost 1.49.0)
Loading boost 1.49.0
Loading epics-base 3.14.12.3
Loading ipython 0.13.1
Loading lapack 3.2.1
Loading atlas 3.8.3 (required by matplotlib 1.1.1)
Loading numpy 1.6.2 (required by matplotlib 1.1.1)
Loading matplotlib 1.1.1
Loading pcds-make 3.2.13
Loading pdsdata 6.1.4
Loading setuptools 0.6c11 (required by poster 0.8.1)
Loading poster 0.8.1
Loading pyca 1.0.0
Loading pyepics 3.2.1
Loading pyfiglet 0.6.1
Loading Minuit2 5.28.00 (required by pyminuit2 1.1.0)
Loading pyminuit2 1.1.0
Loading pyparsing 1.5.6
Loading pythondialog 2.7
Loading qt 4.6.2
Loading root 5.30.06
Loading scipy 0.11.0
Loading simplejson 2.3.2
Loading wxWidgets 2.8.12
Loading xraylib 2.15.0

Thus Minuit2 5.28.00 will be built (if not already built), even though it is not in the release list, because pyminuit 1.1.0 is in the release list.

Each package building class should be a direct or indirect subclass of scripts/pspackage.py. Most will be either a direct subclass of scripts/python_package.py or scripts/unix_package.py. The first is for python packages which are built with setup.py build and setup.py install. The second is for C/C++/etc classes which are built with ./configure, make, make install. If no further customization is needed, a build class can be as simple as scripts/db/Cython.py:

from python_package import python_package

class Cython_package(python_package):
    def init(self):
        super(Cython_package, self).init()

If more customization is needed, then options can be set and methods overloaded. For example, scripts/db/scipy.py:

from python_package import python_package
import psenv
import psrun
import os

class scipy_package(python_package):
    def init(self):
        super(scipy_package, self).init()
        self.build_command = '%s setup.py build' % self.python

    def set_more_paths(self):
        atlas_dir = self.get_absolute_install_dir('atlas')
        psenv.set('ATLAS', '%s/lib' % atlas_dir)
        psenv.set('LD_LIBRARY_PATH', '%s/lib' % self.get_absolute_install_dir('atlas'))
        psenv.set('PYTHONPATH', '%s/lib/%s/site-packages' % (self.get_absolute_install_dir('numpy'), self.python))
        psenv.set('LDFLAGS', '-shared -L%s/lib' % (atlas_dir))

Separate pages explain the various options and methods for the pspackage, python_package, and unix_package classes.

Note:  For help in resolving compilation errors due to missing system software packages, see Resolving System Software Dependencies.