From Core Minutes May 8, 2007
CMT v1r18p20061003 is now in use by RM. Navid and Joanne teamed up to find the problem, which was due to non-backwards-compatible handling of CMTPATH. However, this exercise has demonstrated that we can expect no support from CMT-central. It would be prudent to come up with an alternative which would allow us to dispense with CMT altogether. There are two candidates:
- SCons-based, perhaps in conjunction with Toby's Python scripts on Windows to produce suitable project files. Jim has volunteered to write a Python package would would take our CMT requirements files as input and create SCons SConstruct files, using a subset of ScienceTools to try it out.
- A roll-your-own solution from Navid. He has already done some work on it. He doesn't have sufficient time to put into it at the moment, but likely would in a few weeks.
The plan to eliminate unnecessary environment variables and define necessary ones at run time (not before) has been on hold but is being revived. (Toby) A large fraction of the CMT-generated ones, the <package>CONFIG variables, are unused and could be dispensed with immediately. [Post-meeting Navid has confirmed this. He tagged a newGlastPolicy which turns off the generation of these variables.]
Status from Jim Chiang May 17, 2007
Here is a rudimentary first cut at using python to parse the
requirements files to deploy SConstruct build files through a checkout
package directory tree:
/nfs/farm/g/glast/u33/jchiang/SConsBuilds
The script deploySCons.py finds and reads the requiements files and
writes "SConscript" files in the src subdirectories of each package.
I use the unix find command to discover the req files, but the python
function os.path.walk could also be used to enable portability to
windows.
The top-level SConstruct file is what one uses to do the builds. In
that file, we should put the equivalent of our various policy package
patterns as python functions and Export them so that subpackage
SConstruct files can Import them as needed.
You'll note that at the bottom of the SConstruct file there are the
lines
#SConscript(os.path.join(facilitiesDir, 'src', 'SConstruct'))
#SConscript(os.path.join(tipDir, 'src', 'SConstruct'))
#SConscript(os.path.join(astroDir, 'src', 'SConstruct'))
SConscript(os.path.join(facilitiesDir, 'src', 'SConscript'))
SConscript(os.path.join(tipDir, 'src', 'SConscript'))
SConscript(os.path.join(astroDir, 'src', 'SConscript'))
If you uncomment the first three of these and comment out the last
three, SCons will do a full build using the hand-crafted "SConstruct"
files I placed in each package, whereas using the lines as is will
give you a build using the deployed SConscript files and will only try
to build the libraries.
The astro package req file has a lot of cmt code in it that is not
used in any other package so I decided not to try to have
deploySCons.py attempt to parse and replicate all that logic in
writing its SConscript files. As a result, the astro library does not
build correctly using the Sconscript files.
To build a single package, one simply does
scons <package_name>
from the top-level directory, so we can control the order in which
things are built.
FYI, the perl scripts flattenSAE and cmt2hmake.pl are what Larry Brown
and James Peachey use to create an hmake file for ScienceTools.
-Jim
VRVS Meeting Summary May 23
SCons and Windows support
We should talk to Toby - but the plan would be to create real Visual Studio solution files that people (like Tracy) could really use. Jim mentioned SCons' Builders:
http://www.scons.org/doc/0.97/HTML/scons-user/c2338.html
which could be used to create our Visual Studio files, rather than using the default mechanism that SCons provides, which apparently doesn't produce very useful Visual Studio files.
Note: SCons 0.97 is now installed on the UW TS.
SCons will not require a package reorganization. [I'll stop asking 
]
Both Jim and Navid feel it would be best to avoid parsing our current requirements files, and rather move towards utilizing SCons or the syntax for the NPMS.
NPMS
With momentum building behind SCons should we continue to pursue the custom Navid build tool? Navid provided two remaining concerns about SCons:
1.) We don't know how well supported SCons will be by their development community.
Later Jim mentioned that SCons has a wider community than CMT and one would expect it would work out better than our CMT experience has.
2.) SCons does not terminate a build when a dependency fails - this will lead to problems for RM to report what package failed - we've discussed this in previous emails.
What about Windows?
Navid would provide something similar to SCons' Builder option - to allow users to define Visual Studio files.
Joanne feels we should carry on down the two paths a bit longer and Heather concurs.
Joanne requested that we provide a specification for the command line interface for either choice and Jim had queried Navid earlier concerning his command line interface. Navid responded that it will look a lot like SCons. Navid will try to have something we can try out next week...perhaps a package like facilities and also some package that has other package dependencies?
RM
Navid says any new build tool will require a new RM. He estimates about one month (sooner if a new GSFC hire materializes) to create a new RM. The current RM is patched up for various CMT work-arounds, it would be better to start fresh.
MRStudio
Need to talk to Riccardo - but Heather feels that we should not ask Riccardo to make any major modifications until we are sure what path we are taking - SCons or NPMS.
Update from Navid concerning NPMS May 29
My finger got better over the long weekend. I managed to get my thing
to compile facilities package from science tools. I have taken a few
shortcuts though and I'll be working on fixing those over time. But
technically my program can now be used to tell it to compile
facilities and it'll create the static library and the applications.
The shortcuts I've taken are:
1) When creating the .so and .a files it'll dump them into the
current directory at the moment.
2) When creating an app, instead of searching for the correct paths
to include for the .so and .a files it is right now hardcoded with
the current directory
3) Compile options are not tested. Theoretically they should work
(although somewhat limited) but I haven't tested it yet
4) Which brings up an odd problem when compiling Clock.cxx from
facilities, if you compile it without -g then ranlib complains that
Clock.o has no symbols. It doesn't seem to cause any errors. This
"error" will go away once I have compile options fully working.
I'll continue developing my version unless told otherwise =)
Navid
EVO Meeting Summary May 30th
Joanne Bogart, Toby Burnett, Jim Chiang, Navid G, Riccardo G, Heather Kelly, Tracy Usher
VS2005 and SCons
Tracy has done some investigation into getting SCons to provide real Visual Studio solution files that would be useful for developers. He located where SCons makes its calls to its own Windows build scripts and suggests replacing them with Toby's Python scripts. THis perhaps would only be done for developers, while general users would have no interest in Visual Studio project files anyway, and would use the standard SCons implementation.
Jim asked if anyone has looked into SCOns' Custom Builders and at this point, it seems no one has. It seems to move forward, there should be some further communication between Toby, Tracy, and Jim to see what they can work out for windows.
Tracy had asked via email if we are abadoning the CMT requirements files? Heather mentioned in the meeting that the answer is yes, ultimately.
Toby brought up some things he needs to further understand concerning SCons:
- How to obtain a dependency tree from SCons?
- How to set link and cpp options
- A mechanism to find all packages given a set of paths
need symbols before linking
Dependency Tree from or to SCons builder?
how to define dependency tree
link options
cpp options
setting env at runtime
Dependency Tree from or to SCons builder?
how to define dependency tree
link options
cpp options
setting env at runtime
toby needs:
find all packages given cmtpath
use macros to determine cppflags and includes
Jim
searching paths is cmt
NPMS
st version 8 only statics
Some version available for use?
shared library example?
11:32:33] Navid Golpayegani the binary: /nfs/farm/g/glast/u06/golpa/builderCompiled/builder/src/builder
[11:32:52] Navid Golpayegani checkout of ST: /nfs/farm/g/glast/u06/golpa/ScienceTools-npms
8] Coldstone: but as long as people are OK
with delayed/no implementation of the current RM
feature that tells them which package fails exactly
I can live with it
build order unavailable
need to have scons ignore
failures..then parse output
Jim - astro which depnds on tip
scons tip
scons astro
scons will go back to tip and try to rebuild link
is that the problem
tell cmt to build astro - it only builds astro and
doesn't go off to do something else
Navid can work around that
CMT tells build order - cmt show uses
jim can get dependency tree from scons
slow down rm due to building tip as well as astro
only for linkage
separeate SConstruct files for executables
separate linking
build only object files
MRStudio Upgrade
MRStudio was written to be much more flexible than MRvcmt, so removing CMT is certainly possible. We must retain the package structure of our code. MRStudio uses some CMT functionality:
- CVS for checkouts but uses CMT to get a list of packages for recursive checkouts.
- CMT provides a list of tags via "show tags"
- CMT is used to determine dependencies
Once we have chosen between SCons and NPMS, we should move to get MRStudio updated to provide a parallel path to CMT. Riccardo estimates it would take a few weeks to update MRStudio, assuming we can provide him with details for replacing CMT functionality listed above, as well as how to initiate builds.