Goal

Provide RM builds of GRBAnalysis for distribution.  GRBAnalysis depends somewhat loosely on ScienceTools: the BackgroundEstimator subpackage links to a few ST libraries, and other elements within GRBAnalysis make use of ST binaries.

Method

Create a new container (aka checkout package), GRBAnalysis-scons, consisting of the GRBAnalysis code plus and packages from ST needed at build time.  Do not incorporate packages from ST if they only provide binaries invoked GRBAnalysis.  Assume instead that anyone using GRBAnalysis will have done setup for some ST build and will make use of binaries belonging to that build.

Status

As of June 10th..

GRBAnalysis-scons exists in CVS and contains nearly all source files from the original GRBAnalysis module.  

As of May 27th..

See /nfs/farm/g/glast/u54/jrb/GRBAnalysis-scons.   GRBAnalysis has been augmented with ST packages facilities, astro, tip, embed_python, st_facilities, st_stream and irfs/* as well as infrastructure required for SCons containers.  I've made some minor changes to astro and facilities (committed and tagged) so they will build static libraries in the context of GRBAnalysis (but not ST or GR).  I made other small changes to handoff_response and irfLoader (likewise committed and tagged) to reduce the GRBAnalysis dependence on ST packages to the minimum.

Still to do

  • Add new target type so that arbitrary scripts may be installed in bin directory. (This isn't specific to GRBAnalysis; it will be available to all SCons checkout packages.)
  • Add LATBA subpackage once proper organization is determined
  • (tick) Tag all GRBAnalysis subpackages in the new module (e.g. GRBAnalysis-scons/BackgroundEstimator, GRBAnalysis-scons/LLE, etc.)
  • (tick) Create packageList.txt file for GRBAnalysis-scons enumerating all packages and their tags to be included in a build
  • Add test program(s) Optional, but recommended at least for BackgroundEstimator
  • (tick) Modify cvs tagger program to treat GRBAnalsys container-wide tags as it does ScienceTools and other recognized containers
  • (tick) Create container-wide tags using tagCollector script
    Made GRBAnalysis-HEAD-1-3
  • Add SCons RM database entries in order to automate builds of various kinds (LATEST, HEAD, release tags) on platforms of interest
  • (tick) Finalize organization in CVS and naming.  Propose creating new top-level CVS module GRBAnalysis-scons which will contain everything currently in GRBAnalysis, but with source file names changed to be consistent with standard conventions (e.g. .h rather than .hh; .cxx rather than .cc). Also add sym links in the repository for ST packages to be included.
    As of June 2nd GRBAnalysis-scons has been created in CVS but so far contains only BackgroundEstimator, BayesianBlocks and containerSettings, plus sym links to some ST packages and to SConsFiles. This is sufficient to build the BackgroundEstimator library.
    As of June 10th all subpackages have been added.
  • (tick) Add SConscripts for GBMtoolkit, GtGRB, etc.(all  subdirectories of original GRBAnalysis  which don't yet have an SConscript)
  • Determine exactly what needs to be built and installed or just installed; update SConscripts accordingly. See section "Input Needed" below for more detail
  • (tick) Test that all build products get properly built and installed and can run when user has set up environment as intended.  That is,
    • set environment variable INST_DIR to root directory of ScienceTools build
    • source $INST_DIR/bin/redhat../_setup.sh (or .csh)
    • set environment variable INST_DIR to root directory of GRBAnalysis-scons
    • source $INST_DIR/bin/redhat.../_setup.sh

Input Needed

For each subdirectory of GRBAnalysis I need to know what needs to be built (also how, in some cases) and what needs to be installed in order to organize the files according to standard conventions and write a suitable SConscript.

BackgroundEstimator (owner: Vlasios) This one is in good shape already.  It builds one ROOT library. SConscript is ok as it stands.

BayesianBlocks (owner: Jim) Has only one python file.  I have moved it to a directory called python for consistency with other packages.  Had no SConscript but it was easy to write one.

CompositeLightCurve (owner: Nicola?) I need to understand the contents in order to be able to organize it properly.  Is everything in there still of interest?  Should all the .py files be installed?  There is a file BayesianBlocks.py in this subpackage.  How does this relate - or does it - to the file BaysianBlocks_python.py in the BayesianBlocks subpackage?

GBMtoolkit (owner: Giacomo)  Would like to make subdirectories src (for .C) and python (for .py).  .so file does not belong in CVS. What command or commands are used to create the .so?  This will need to be in the SConscript. Typically an .so is specified as some type of library target and is installed under the top-level lib directory.  That directory will be included in LD_LIBRARY_PATH for users who run the _setup file. 

Should all the python files be installed?  If so, maybe should rename ftools.py to something less likely to cause confusion or conflicts. Once all this is straightened out, should be straightforward to make an SConscript.

GtGRB (owner: ?) Would like to make subdirectories src (or perhaps some other name, depending on how .C files are to be used) and python.  What happens to the .C files?  Should they be built into ROOT shareables or are they invoked as is, in ROOT sessions? Why are python files spread across several directories?  Should there be a GTGRB subdirectory of the python directory for the GRGRB module?  Either I need to understand much better how all these files are used, or someone who knows them has to learn something about how SCons operates.

LLE (owner: Frederic) Usual comments:  move all files except SConscript, .cvsignore out of top directory.  Determine what needs to be built; what needs to be installed. The SConscript and file LLELib.py in there now build the LLE library. Will have to add to SConscript if other files belonging to the package need to be installed.

vFvCL (owner: Nicola) There are only two files which, at most, get installed so there can't be much work to do here, but, with no idea what these files are for, I can't do it.

A Plea

There are several files throughout GRBAnalysis with no comments whatsoever.  Usually, but not always, the name of the file is informative, but this really isn't enough for anything you expect other people to use.  Please add a sentence or two at the top, explaining briefly what the file is for, inputs and outputs and other similar information.

Old Stuff

Most likely obsolete. I'm preserving this text in case we decide there is some value in an STExt, either for GRBAnalysis or in some other circumstances.

Method

  1. Make a ScienceTools external library.  That is, for current ST tag and others in the future as needed, install the ST build (libraries, binaries, includes, python, and any other public files as needed) in the SLAC externals area, rearranging the file hierarchy somewhat to be consistent with norms in the Externals area. Add entries to allExternals.scons for this new external (known as STExt)
  2. Re-organize GRBAnalysis into standard form for a container (aka checkout package).  It will depend on STExt as well as on all the externals which ScienceTools itself depends upon.

STExt

I've written a script and committed to CVS (grits-tools/python/InstallST.py) to be used to install the STExt external at SLAC for all supported Linux platforms.  Here is the help:

jrb@noric10 $ python InstallST.py --help
Usage: InstallST.py version-to-install [options]  

Options:
  -h, --help            show this help message and exit
  -a ARCH, --arch=ARCH, --architecture=ARCH
                        32 or 64  [default:value for log-in machine]
  -o OPSYS, --os=OPSYS  rhel4 or rhel5 [default:value for log-in machine]
  --opt, --optimized    use optimized build of ScienceTools [default] 
--debug, --noopt   use debug build of ScienceTools
  --parent=PARENT       parent directory []
  --fake                print proposed actions but don't execute
  --real                really do it
I've made a rhel4 32-bit STExt external for the 09-23-01 tag and have installed it in the central GLAST_EXT area. See /afs/slac/g/glast/ground/GLAST_EXT/redhat4-i686-32bit-gcc34/STExt/09.23.01 (Externals aficionados may notice that the extra compiler level - gcc34 in this case - has been omitted from the directory structure.  This was inadvertent at first, but I left it alone since there seemed to be no harm in it. The entry in allExternals has been written consistent with this organization.)

InstallST.py is now working (as of May 10).  I used it to install the 64-bit rhel4 and rhel5 builds of 09-23-01.  Could not install rhel5 32-bit because the GLAST_EXT disk for that platform was too full already.

STExt has all the installed libraries, binaries (executables; not wrapper scripts), includes, python files, data files, and system pfiles from ScienceTools-09-23-01.  Is this the right set of things?

SCons Description of STExt

Since GRBAnalysis only requires a small subset of the ScienceTools libraries (facilities, tip, astro. *Respose, irfInterface, irfLoader and rootIrfLoader) I defined a library group in allExternals.scons consisting of just these libraries called STforGRBLibs.  The SCons configure step fails, I suspect because of the mixture of shared and static libraries. It is almost certainly an artifact of the particular way the configure check is done, and not because anything is actually wrong with the libraries. For the time being I've added a new type of key in allExternals.scons allowing the definition of a library group without doing the configure check.

  • No labels

9 Comments

  1. Nicola Omodei

    For GtGRB (for which I am the owner).
    There is nothing to build, it's just python. Files have beed divided into several sub-directories (GTGRB, app, scripts) just for logical order.
    Let me know if it is more desirable to have only one python directory with everything inside.
    This require some work, but I can certainly do that in one-two days...

    For CompositeLightCurve: I haven't thought about what we need to install. GtGRB uses some code from CompositeLightCurve, loading this module in python.
    BayesianBlocks should be the same code as the BayesianBlocks module, but I need to check and to change the import statement.

    1. Joanne Bogart

      For GtGRB I suggest we make a python directory with subdirectories preserving the old organization, only everything would be one level deeper.  I don't mind doing this along with other reorganization and renaming (of .cc to cxx and .hh to h) throughout.

  2. Unknown User (vlasisva)

    Regarding the LLE folder: it is a superposition of different codes from various people at various time. It definitely needs reorganization.

    Fred's code comprises of LLE, src, LLELib.py. The other files are mostly Jim Chiang's LLE-plotting code and Giacomo's LLE digger code.

    http://www-glast.stanford.edu/cgi-bin/viewcvs/GRBAnalysis/LLE/?root=CVS_SLAC

    For gtgrb, it is becoming a large code so having everything dumped in a common "python" folder will make things less tidy.

    1. Joanne Bogart

      It's perfectly all right to have whatever substructure you want under the top-level python folder. We could name subfollders after the current collection of top-level folders. Would that be satisfactory?

      1. James Chiang

        For the LLE subdirectory/subpackage, I think it would be fine to move those python files to a python subdirectory under LLE. We now have a ScienceTool executable that will do some of the work in those scripts, i.e., making the LLE data selection from the merit file, so those scripts will need to be updated anyways, and some will be eliminated entirely.

        1. Joanne Bogart

          In a working directory I've tentatively rearranged LLE as follows:

          • put all .py files (except LLLib.py) in a new python subdirectory.  Should all of these be installed? Some names (e.g. read_data.py) could collide. Could rename, move to a subdirectory, or not install.
          • moved grb_par.txt to a new pfiles subdirectory.  Should this be installed?
          • moved all .C files to a new subdirectory src/macros

          For the time being I left setup.csh and setup.sh alone.  They will need revision for the new arrangement (will be simpler, I hope).

          Once these issues are settled I will add LLE to GRBAnalysis-scons.

          1. James Chiang

            I would have SCons install all of the .py files in a ${INST_DIR}/python/LLE subdirectory. We can rename any files that have conflicts.

            grb_par.txt is meant as an example file, so it doesn't need to be installed.

            The setup.[c]sh files don't need to be ported to GRBAnalysis-scons. The standard _setup.[c]sh files should do the trick.

  3. Unknown User (vlasisva)

    For (my) reference, this is how I compiled at SLAC

    ssh rhel5-64.slac.stanford.edu

from csh

alias scons /afs/slac/g/glast/applications/SCons/1.3.0/bin/scons
setenv GLAST_EXT /afs/slac/g/glast/ground/glastsoft/AUTO_GLAST/arch/os/GLAST_EXT
setenv INST_DIR /afs/slac/g/glast/ground/glastsoft/AUTO_GLAST/arch/os/SCONS_BUILDS/Optimized/ScienceTools/09-23-01
source ${INST_DIR}/bin/redhat5-x86_64-64bit-gcc41-Optimized/_setup.csh
cvs co -D GRBAnalysis-scons GRBAnalysis-scons
cd GRBAnalysis-scons
scons --with-GLAST-EXT=$GLAST_EXT --cxxflags=-ggdb --ccflags=-ggdb --compile-opt

    1. Joanne Bogart

      It shouldn't be necessary to source the ScienceTools _setup in order to build GRBAnalysis-scons though it shouldn't hurt, either.  It is necessary at run time.