SCons Status 2010

August 2, 2010

As SCons Turns

Attendees:  Joanne Bogart, Toby Burnett, Jim Chiang, Heather Kelly, Tom Stephens Tracy Usher

Snow Leopard

Tom has been working to get the automated builds going but has run into problems.  Attempts by the SCons build to write to u35 fail.  When re-run using a local area, the SCons build completes successfully.  SCS admin, Yemi, has been working with Tom to solve the problem.  First attempted fix was associated with resetting some sticky bits associated with the u35 directory.  That did not end up solving the problem.  Yemi has been set up to attempt to run SCons himself, and the last Tom heard, he was trying to do a clean, so he could re-run the build.  In the meantime, getting the VC90 builds going is a higher priority and Tom has turned his attention there.

Pick a copy of source, any copy of source

To address the development issue of updating the appropriate copy of source and includes so they are picked up at build time, Joanne has been working with setting SCon's duplicate setting to zero - which turns off the variant copies of the source.  Builds work just fine for vanilla packages, issues arise with package that run rootcint which create source files and is sensitive to where it is run, and where it stores its output.  Joanne has tried out copying the include files in advance for st_graph and that worked.  Joanne has also sent mail to the SCons discussion list to see if there is a neater way to handle this.  On the linux side, it seems we can set this up to use hard or soft links for the copy of source.  Jim suggested that we should use soft links so the copies are kept in synch. 

Toby reported on a problem he's seen in Visual Studio where the project files utilize conflicting versions of the includes.  The local copy of the includes are used in the project, but the include directive  points at the top-level installed copy of the includes.  Consequently, one will update the wrong copy of the includes.  A solution would be to update the include directive to prepend the local path to the includes.

Action:  Joanne will finish up with duplcate setting and will take a look at windows.

Trouble with tip

As Toby has reported via email,  there are a couple of problems with tip in VC9  He was able to work around one of them, where the iterators failed some error checking.  The other issue occurs at runtime where tip attempts to determine if an input file is FITS or not.  James Peachey has been alerted and assures us that he is still 30% Fermi.  One issue is making VC9 available to him so he can see the problem.  We agree to set up UW TS for this, and Toby has done that and sent mail to James detailing how to reproduce the issue.  James Peachey also reported his progress with the CFITSIO upgrade, where tip had some trouble with cfitsio v3250.  Bill Pence has provided one fix.  There is a remaining issue for James to work on.  For now we agreed he would focus on fixing up the window VC9 issue.

Automated VC90 Builds

Tom may have worked out the remaining issue with the autmoated VC90 builds, where the batch environment was not being set up fully.  Ultimately, Tom may provide a bat file to run to set up the environment, for now he has updated the glastlnx20 env for the glast account.  Hopefully VC90 builds will be ready by the end of the day.

Recent tags not reflected in ST SCons LATEST

Toby reports that recent tags do no seem to be making it into SCons ST LATEST builds.  Tom suggests Toby provide a concrete example and he will check into it.

pyd vs dll

As Toby mentioned via email, we need pyd extension for python import libraries.  Heather was unclear whether we need to retain the dll copies as well. Toby assures us that we do not.  For now he can work around this issue by renaming the files by hand, but ultimately we need to update the swig builder.  He wonders where this is handled?  Joanne believes this is in processExternals, and will send him the details by email.  Then Toby can make a local fix and pass the update to Joanne who will commit it to CVS.

Embedding Manifest Files

 While we already create manifest files, it seems we need to embed them into the libraries.  There are instructions for how to do this on the web.  The crucial bit to update is probably in processExternals in SHLINKCOM, to update the swig builder.  Alternatively, Joanne would like to look into chaining builders which may be a cleaner implementation.  Toby mentioned a desire for more granularity and a walk through of the existing interface we have introduced on top of SCons.  Joanne mentioned that most of the details can be found in the SCons manual and to start there.

/TP flag in Visual Studio projects

Toby has discovered that the SCons command line is slightly different than the settings in the resulting VS projects.  The VS projects include a /TP flag which forces all source to be treated as c++ code.  This is bad for the astro package where there is true c code in there.  Joanne suspects a bug in the project creation step and will look into it.

Questions of Dependence

Toby also noted that when building a package like pointlike which depends on skymaps, it would be better to have the swig wrappers in skymaps built too.

bindexplib

After reflecting on Tom's email we opted for door #, where we use bindexplib and hoops again creates a static library.

Agenda

• Development and multiple copies of the source
• Windows
  • Trouble with tip reading input FITS files
  • pyd versus dll
  • embedding manifest files
  • extraneous compiler flag /TP that forces .c files to be treated as C++. This causes the astro/wcslib to fail
  • SWIG (and unit test) build is not run unless the package is explicitly a target
  • allExternals obtained via installer versus from CVS
  • Turning on automated VC90 builds
  • vc71 builds and bindexplib
  • Moving Forward
• Identifying builds in the installer:  LATEST-1-3206 versus LATEST-3206.
• Snow Leopard

May 12, 2010

Attendees:  Joanne Bogart, Jim Chiang, Heather Kelly, Tom Stephens

Preparing for Termination of ST CMT Builds

ASP

Jim has tested in DEV, though he points out that doesn't guarantee problem-free running in PROD.  We'll know for sure when we try it out.

RHEL5-64

64 bit builds are working as well as the 32 bit ones.  The main problem was with the pulsar tools and Masa helped to sort that out.  We might want to send a note to softlist to notify people that they can try these builds out.

Actually Jim had a question about 64 bit builds - on his machine, it seemed that the build defaulted to 32 bit, and he wanted to know how to turn that on.  Joanne replied that she's had the opposite issue, in that, she had to specify 32 bit on her rhel5-64 machine.  There is a flag to pass to SCons, --64bit that should take care of this.  Jim will try it out.

Other Items

bindexplib and hoops

James Peachey gave the go ahead to build hoops as shared.  Tom has made that update in CVS and removed the #ifdefs that Joanne had in place for her workaround.  So far so good.  We're still using the ROOT supplied bindexplib, we could still move to Tom's python script.

stag

Tom believes the run condition is fixed and is waiting for confirmation the next time the situation arises.

deleteBuild

Tom took a look at it, it appears to clean up the disk space and remove the entries in the DB for the location of user and dev releases, but the actual entries in the DB remain.  This means that when we go to retrigger, the system will see that the entry already exists and reject the request.  Tom wants to consider how best to proceed, so we delete the entries or provide a new field to denote that the build is no longer active.

Turning on windows head and release builds

Tom is looking into this now.

Email notification of build failures

Tom is looking to see where this bit occurs in the SCons RM.

April 21, 2010

Attendees:  Joanne Bogart, Heather Kelly, Kim Lo, Tom Stephens

CMT RM

Kim has set up a shadow table to keep track of deletes.  He discovered that the CMT RM DBs are in MySQL 4.1.22.  This version apparently does not support triggers.

Joanne reminds us that we can detemine the version of the server via:  mysql> select version() ;

ASP

No News

Gaudi upgrade

Heather continues to look at rootmap files on windows.  At the moment, the visual studio project is being created, but no output rootmap file is produced.  Heather sis studying the fragments.

RMViewer

A new version will be tagged by Joanne, to provide better access to the logs.  This can be done in Karen's web  pages, but it's nice to have in RMViewer too.

ST

SCons 1.3.0 upgrade

We seem to have migrated on Windows after making sure to update the scons.bat files on the V: drive.  Next step will be to move linux and Mac, once we have SCons 1.3.0 installed under $GLASTROOT.  Joanne also reports that there is talk of SCons 1.3.1, though it is unclear when that would  become available.

bindexplib replacement

There are three options:  Tom's python script that uses dumpbin from VC++ to export symbols.  Take bindexplib source and maintain it ourselves, Tom has a moodified version that comments out the section that checks for declspec.  Or continue to use bindexplib as packaged in ROOT.

Tom has been trying out various combinations of his new tool written in python, ROOT's bindexplib, and a Tom modified version of bindexplib on hoops.  Tom believevs we should update hoops to build as shared and will contact James Peachey about that.  Joanne will go back and look at her notes associated with her attempts to build as shared on windows.

Tom will look into the RHEL5-64 builds.  Joanne reports that actually, the RHEL4-64 builds seem to need some help.  After the meeting Joanne found that the GLAST_EXT area for hte new ROOT (v5.26.00a-gl1) was not named correctly, causing the builds to fail.  That has since been fixed.

command line installer

Tom has added a logging facility and some increased debugging statements when doing the transfer in response to David Landriu's problems obtaining the user releases.  Both Heather and Tom report no troubles using the command line installer, but admittedly they are nearby transfers.  Most of our previous problems have been long distance, such as across the pond.

April 14, 2010

Attendees:  Joanne Bogart, Heather Kelly, Kim Lo, Tom Stephens

CMT RM

Kim reports that there is no record of deletes, and would like to create a shadow table to keep track

ASP

No news

Gaudi Upgrade

Heather continues to look at rootmap files on windows

April 7, 2010

Attendees:  Joanne Bogart, Jim Chiang, Heather Kelly, Kim Lo

CMT RM

Kim has completed his work on the dependency database so that Tony may perform queries to find valid builds for installs.

 
The installeMap table is populated using the existing build directories.  There are still some discrepencies in that some directories do not correspond to entries in the checkoutPackage table.  This is likely due to failures to delete those directories from the deleteBuild script.  Navid had noted in the past that this could happen.  Next Kim will look at modifying the createInstall and deleteBuild scripts to keep the tables in synch.  Heather will send Kim any notes she finds from Navid concerning the deleteBuild script as well as the location in CVS to update those scripts with Kim is finished.
 
GR

Heather has finished off her work on Wired and GR, and can now return to Gaudi.

Joanne has some remaining updates for GR packages, which may be related to job options.

ASP

Jim reports that he has tested ASP and it seems to work.  Now he is trying to find the right moment to test it in production.  ASP accesses database tables, and there is only one instance in production.  Before testing Jim will circumvent those parts.

SCons

Tom has completed a beta version of a command line installer script written in python.  We agree it's ready for beta testing, and Heather will send mail to David Landriu, Jean Ballet, and Jim Chiang.

Joanne is waiting for a response concerning her trouble with the new SCons 1.3.0 and choosing the correct version of Visual Studio.  There have been some other checkpoint releases to address other Windows-related problems.

Joanne has some RMViewer enhancements she'd like to complete at some point.

 

March 31, 2010

Attendees:  Joanne Bogart, Heather Kelly, Kim Lo, Chuck Patterson, Tom Stephens

CMT RM

Kim is working to further understand the dependency DB and how we might keep it updated so that the installers and any scripts using the dependency DB can know that a build exists or not.  At the moment, deleteBuild does not update the dependency db at all, due to a problem Navid had in trying to get it to erase entries correctly.

Kim requests more details concerning the fields in the dependency database, such as mainPkg (which we think is used to denote whether a package is a checkout package like GR or ST), as well as details about foreign keys and constraints in the ReleaseManager DB.

SCons

Joanne reports that SCons 1.3.0 is now available.  She has tried it on windows, and it seems a problem with SCons determination of what versoin of Visual Studio to use is still there.  It may be associated with cloning environments.  Joanne has sent a note to SCons developers and itseems there is now someone actively listening and addressing windows problems, so that is good news.  Apparently SCons 1.3.1 is planned, as well as a 2.0 which will break compatibility with older versions of python (pre-dated 2.5.1 which we are currently using).

GlastRelease

Heather needs to get back to Gaudi.  Currently finishing up modifications associated with the conversion services.  Next up is to address the changes in Gaudi where on windows they now use *.rootmap files to handle dynamic loading of libraries, to replace the older mechanism of *_dll.cpp and *_load.cpp files.

Science Tools

Chuck requests a poke when the new ST tag is made available.  Jim has promised one for this week and that will automatically trigger a new SCons build.  Heather plans to take a peek at the current ST LATEST builds to see that things still look ok.

Heather also plans to continue looking at RHEL5-64 builds to get that finished up.  There remains some unit tests in need of attention.

SCons RM

Tom has found one issue associated with the failure to create or update the DB for user and developer releases.  The introduction of the  jobOptions directory into the tarball command causes trouble for the ST builds, since there is no jobOptions directory (and there is no need of one).  He and Joanne are working out updates to the SConstruct file to work around this and still support a jobOptions directory for GR.

Tom has created a windows replacement for bindexplib.  Currently, his program creates a def file identical to that produced by bindexplib.  Next up according to Joanne is to refresh their memory about what goes wrong sometimes, and determine what changes we can make to our program to get it to do what we want.

Tom is looking to create a command-line installer script modeled on the CMT command-line script, but using the java DB interface that Tony J provides.

Tom updated the triggerBuild script last week, as well as the RM DB tables so that we could trigger more combinations of builds.  Heather reports that the deleteBuild script still does not work, though it reports that it is indeed scheduling a build for deletion, nothing ever happens.  Tom will look into this when he can.