SCons thread started by Toby Burnett:Joanne made a very strong statement to the effect that CMT was very badly designed. I regret not following this up at the time, and I'd really like to hear her analysis, so that we can consider how we can mitigate the effects, or find a proper replacement. As I said, I'd like if we could consider scons, [ http://www.scons.org/] ,but I don't have the time to really investigate it myself.
 

Followup by Richard Dubois:

 Before we go far, if anywhere, with scons I'd like to see a discussion from all the stakeholders on what a changeover would entail - how it would affect RM, the development and build tools, etc, etc. I would prefer we don't just change horses to end up with a different set of problems...

Richard

Followup by Jim Chiang on 3/6/06:

As part of the feasibility exercise, I have implemented the scons build for facilities and imported the package into my user area as scons_facilities: http://glast.stanford.edu/cgi-bin/viewcvs/users/jchiang/scons_facilities/ After checking this package out, cd to the src directory and type "scons":

glast-guess1jchiang scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o Clock.o Clock.cxx
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o ScheduledEvent.o ScheduledEvent.cxx
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o Scheduler.o Scheduler.cxx
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o Timestamp.o Timestamp.cxx
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o Util.o Util.cxx
ar r libfacilities.a Clock.o ScheduledEvent.o Scheduler.o Timestamp.o Util.o
ranlib libfacilities.a
Install file: "libfacilities.a" as "/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4/rh9_gcc32/libfacilities.a"
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o test/testUtil.o test/testUtil.cxx
g++ -o test/testUtil test/testUtil.o -L. -lfacilities
Install file: "test/testUtil" as "/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4/rh9_gcc32/testUtil"
g++ -I/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4 -c -o test/test_time.o test/test_time.cxx
g++ -o test/test_time test/test_time.o -L. -lfacilities
Install file: "test/test_time" as "/scratch/jchiang/ST/SconsTest/scons_facilities/v2r12p4/rh9_gcc32/test_time"
scons: done building targets.
glast-guess1jchiang

Copies of the library and test program will be placed in the rh9_gcc32 directory on linux and in VC8debug on Windows.
This package builds correctly on slac linux and on glast-ts, where I have installed scons as part of the Python 2.4.1 distribution. To clean, just enter "scons -c".
A couple notes:

• The SConstruct scripts in the src and src/test subdirectories would be modified substantially as one adds layers in the build hierarchy. In particular, the "Construction Environment" objects in Scons play the role that the policy packages have in our use of CMT, so some of the code that sets compiler flags and so forth will move up in the
  build hierarchy for more realistic examples.

• SCons calls the compile and link commands directly on each system, so that means that on linux there are no Makefiles and on Windows there are no project or solution files. I am not sure what implications this will have for MRvcmt or MRStudio, but at the end of the day, I expect that the architecture of those programs would likely be simplified substantially if SCons were used.

I'll tackle tip next, its being a package that depends only on external libriaries, and then after that, astro, which I believe is the simplest non-trivial package in terms of dependencies.

-Jim

Follow up by Navid Golpayegani on 3/6/2006:

A few disclaimers first:
I don't mean to sound negative since I actually would love to see  us move away from CMT. This is not directed directly at you. It's actually a great first  step. This just happens to be the only one I still have to reply to  in this subject.

Anyways, I think we are going about this the wrong way. Converting  Scons to create the same thing as CMT does little to help. We should  be looking at how Scons (if at all) can help us improve (by adressing  the faults of CMT).

One of the biggest flaws of CMT (or our way of using it) is that when  you have multiple libraries/applications listed in your requirements  file you can't easily make them depend on each other. Our attempt at  solving this is to use static libraries (which are built before  shared ones) and to separate out test apps into their own packages.

CMT links against unnecessary libraries. When you depend on another  package, you automatically inherit all the libraries that package  makes and all its sub-packages. Again this is not desired all the  time. A variation of this is that if your package creates two  constituents and one depends on a different package and the other  does not. Both your constituents have no choice but to use this other  package.

There's more flaws in CMT (redundant versioning, container packages,  etc..) but the above two are recent ones that are biting us over and  over again that are also major flaws I think.

I think we should look how Scons would handle these problems first.

Navid

Followup by Jim Chiang on 3/6/06:

No problema. I share all of the concerns regarding the way CMT behaves that you have given below, including a few others. One of the things that SCons does exceedingly well is the working out of cross-package dependencies. I think its capability of doing that well is at least as valuable as the ability to use the power of a full scripting language to implement the logic of the builds.

It is probably worth reviewing the SCons page to see what it is capable of: http://www.scons.org/
 

Followup by Navid Golpayegani on 3/6/06:

I've been looking over the user manual after I sent my email. I think  I can say with some certainty that Scons would address those problems  with CMT.

Another thing to promote it is that Jim Chiang's request (and I'm  sure others would like this too) to have wrapper scripts built during  compile (as opposed to having the RM generate them) is potentially  easy with scons. The reason is that it's a full blown python language  and you can therefore have logic in there.

Of course to be certain I'd have to try it out which I'll do when I  get the chance (not going to be soon).

Navid

Followup by Toby Burnett on 3/6/06:

Well, I'm delighted about the momentum that is building for looking at scons seriously. But I have one comment with respect to Navid's point about dependencies. CMT takes the "use" statements seriously, which often are redundant, or even superfluous. Then there is the confusing choice of whether to put the use statements in private sections, or to qualify them with "-no_auto_imports". A system that discovered the actual dependencies, by examining the include statements in the files for example, is far superior. If we stick with CMT, we would really need someone to police dependencies, and remove improper ones.

But I must say that my attempt to fulfill this role was a disaster, making some enemies.

-Toby

Followup by Jim Chiang on 3/7/06 (apologies... there are some lost messages in the thread at this point):SCons calls the compile and link commands directly on each system, so that means that on linux there are no Makefiles and on Windows there are no project or solution files. I am not sure what implications this will have for MRvcmt or MRStudio, but at the end of the day, I expect that the architecture of those programs would likely be simplified substantially if SCons were used.

Followup by Tracy Usher on 3/7/06:

Does this mean it would NOT be possible to teach scons to make project/solution files? Or would this become the purvue of MRvcmt/Studio? I'd worry if we had one system to do RM style builds and another system for development. But I don't claim to have looked at how this system works at all so might be missing some key points.

Followup by Jim Chiang on 3/7/06:

According to the SCons page, SCons has "Built-in support for Microsoft Visual Studio .NET and past Visual Studio versions, including generation of .dsp, .dsw, .sln and .vcproj files."

Followup by David Chamont on 3/7/06:

I would also love to leave CMT lands for new exciting countries such as SCONS, but I do not expect
the switch to be simple and fast. As said by richard, we will change horses to end up with a different set of
problems.

Yet, there is much work and enthusiasm around SCONS, so it is very worth to try some
prototyping work such as the one started by James.

On the side of CMT, I am convinced we can get the CMT developers top improve CMT the
way we want, or at least to have the feedback so that we can better use the tool.

The first important step is to describe very precisely the problems we think we have.
Is there a confluence where to list those problems ?
If not, we should create one where to centralize all the recriminations on CMT.

Other comments below :

Riccardo Giannitrapani wrote:

Hi Richard

well, you know, I expected this .. I am not sure we will get so much  advantages in leaving CMT, but I'm not fighting every day with RM and  things like that, so I can understand Navid point of view .. from my point  of view (gui)

- MRvcmt is completely linked to CMT, no way to remove it and replace with  something different

- MRStudio has been designed from start to not depend directly on CMT ..  cmt is a module (not yet complete, I'm really slow in this period, I know)  and in principle can be replaced with another module to handle building of  packages .. but there are two issues
  - MRStudio is designed to work on a packages structure like the one we  have now .. so that structure should be preserved .. this is something I  like of CMT (maybe one of the few good things), the way it forces the  design of software in separate packages with dependencies .. I really  would like to see this kind of structure also with other tools
  - We need a completely equivalent mechanism of "cmt show uses" to be  able to build VS solution file .. moreover the solutions files depend on  the build of the .vcproj files that again are built by cmt .. if scons  does not allow this, we will need to build them, that maybe is trivial,  but for sure is something to be done if we want IDE support

What "generic" and "independant" MRStudio has been designed, if CMT is the only real build tool which has
been connected to it, be sure that when you will try to connect whatever other tool, you will discover that
your interface is still not generic enough, and that you implicitely used cmt-specific behavior here and there
in MRStudio.

For the rest MRStudio just invoke system command to interface to cmt and  that can be done also with scons, I presume .. the problem now can be the  time schedule .. I was quite relaxed on this since we have MRvcmt in the  meanwhile (for example in the last 10 days I've worked more on FRED and  some related issues), but if we are going to abandon CMT soon, I think  MRStudio need a burst from my side .. and I will need also to learn scons  ..

Last question; why scons? Is the first one or do you have evaluated other  building tools and decided for this? I have nothing against scons, just  curious ..
This is the new build tool which seems to make developers enthusiastic.
If I remember well, it is in python, and surely this explains part of its success.

I remember I've looked into scons some time ago (for another  project) and didn't found a simple way to support dependencies the way CMT  does between separate packages, it seemed more suitable to build full tree  of software directories .. but I've not looked at it in details, so I'm  sure I'm wrong ..

Other thing: is not possible to contact CMT makers (I remember Toby know  them) to see if it is possible to ask some changes the way CMT work to  alleviate our problems? Or maybe they can suggest a different way to use  CMT to solve them as well .. I know, this is maybe naive, but before  leaving a working (althought problematic) system one should try to fix it  .. maybe we are just using CMT in a wrong (or not perfect) way .. for  example, I see a package as an atomic object in CMT design .. I think that  if in the same package two constituents uses different dependenecies, this  is not CMT problem but point to a need of reorganization of the packages,  maybe these two constituents should not really belong to the same package  (but again, I'm not an expert on this, so maybe I'm completely wrong).

Surely the pragmatic best way. The former author of CMT is now lost in other responsabilities, but
there is a new frsh engineer working on CMT full time now (as far as I know).
Yet, for each of our problem, surely we should state it clearly, and ideally to prepare
a demonstration of it.
This is why I suggest a new confluence page for this.
Also, the same as Toby, I am very much interested if Joanne can
explain why she think cmt is "bad designed". On my side, I
would rather say "too complex", but if joanne can point out
some concrete bad-design-case, it would certainly help.

David.

Followup by Toby Burnett (response to Visual Studio concernts above) on 3/7/06:

 I worry a lot about this because that is what we have now, with Navid's scripts running independently of the visual studio solution setup. If
all building is defined by one set of files, the SConstruct files associated with each package, maybe builds would be consistent.

Another problem is that the RM/Install setup is not necessarily consistent with the usual developer's use case, which is to install a
release, and then check out packages to develop against the release.

--Toby

Followup by Tracy Usher on 3/7/06:

Perhaps of interest....

http://www.gamesfromwithin.com/articles/0509/000100.html

Followup by Jim Chiang on 3/7/06:

 A timing test on my desktop machine:

glast-guess1jchiang time glastpack.pl rebuild astro v1r13p7
<...snip...>
test ok.
183.930u 32.180s 3:44.51 96.2% 0+0k 0+0io 1292659pf+0w

glast-guess1jchiang time scons -c
<...snip...>
scons: done cleaning targets.
2.640u 0.300s 0:03.57 82.3% 0+0k 0+0io 3917pf+0w
glast-guess1jchiang time scons
<...snip...>
scons: done building targets.
106.920u 6.150s 1:59.29 94.7% 0+0k 0+0io 109205pf+0w
glast-guess1jchiang

The winner: SCons.
 

Followup by Tracy Usher on 3/7/06:

They do make this claim... And there is a nice looking "MSVSProject()" function available and sort of described in the "man page" (which I thought meant "manual" but am now thinking has a slightly different meaning...).
I took the example SConstruct file in Jim's user area for the facilities package (which, conveniently, doesn't have any dependencies) and tried to see if I could make a project file. Eventually, I did succeed (I've attached the results to this e-mail) though I could never get these files to go to the package "Visual" subfolder or get the resulting library to go to the "VC8debug" folder, or get the object files to go to the right place either... Ok, I'm sure these problems could be easily solved if I had a bit more Python experience. Oh, and I've also not yet been successful at including the test program (so can't yet comment on debugging issues that might arise).

Anyway, if you look at the facilities.vcproj file the one thing to not is that the actual "building" is all done in scons macros, which means there is no access inside visual studio to compiler options, include paths, etc. We do currently build dlls via a macro (which I would like to see changed as well), but its often been REALLY REALLY useful to have access to the compiler options (e.g. just recently with the new external libraries) that we do get with our current cmt based system.

IMHO, if we are going to go this route then we need to put together a test of multiple packages, tying in external libraries, and make it available to developers so they can make sure they can still "easily" develop and debug their code in the currently supported environments. I'm sure this is a much bigger issue to the few windows developers out there, but from where I am sitting right now I'm worried that this will be a big step backwards unless we are willing to invest real time into beefing up the python scripts.

Tracy

Followup by Navid Golpayegani on 3/7/06:

I agree compile time is not an issue for us. Besides I think we'll be  making a move towards improving our compile time. I have a suspicion  CMT is slower than all of those.

navid

Followup by Toby Burnett on 3/7/06:

 Well, looking at the vcproj file, thanks for making one for us, I see your point, that it uses scons to perform the build actions. That is a good thing, I think, in that in forces consistency.

To modify the options, which I often do myself, they conveniently provide the Sconstruct file in the project. So that might be enough flexibility. In fact, it might be better than our current practice in one respect: whenever we change the build environment, we have to go back to MRvcmt, or CMT, and remake the solution files.

What it would not do, which I like very much in our current scheme, is to generate a big solution containing the current project, as well as all the dependent projects. But then that would be not a big deal for an script, as is currently done in MRvcmt. Bottom line to me, is that things like that get a lot more accessible when it is being done in
python.

--Toby
 

Followup by Jim Chiang on 3/7/06:

 FYI: The slightly more complicated example of building the astro package is here:

http://glast.stanford.edu/cgi-bin/viewcvs/users/jchiang/SConsBuilds/

You will need to go to the top directory and type
% scons

to build the whole thing, or

% scons facilities
% scons tip
% scons astro

to build individual packages. I haven't sorted out how to get the build to go from the individual package directories. It's looking like we may have to give up on that .

In any case, the overall build works well on linux, but I am encountering a problem with dlls on Windows that a Windows expert may be able to figure out....

Feel free to play with this and check in your mods especially if you manage to get the Windows build to succeed.

-Jim

Followup from Tracy Usher on 3/7/06:

Thanks, that is another good example.

I pieced together a solution file that let me open the facilities project with its two test programs and I tried to run in the debugger. No good, even though I added debug switches explicitly to the build the code appears to be without debug symbols...
At this point I'm accruing more problems than I'm solving so I'm going to head home, tail between my legs, while I still can!

Well, on the plus side I can run the text program from the command line and it does appear to run successfully. Just no debugging...

Tracy

Followup from Tracy Usher on 3/7/06:

There are occasions when you want the full solution file which includes all of Glast... For example, it was a lot easier to track down all pieces of code that needed updating for CLHEP with the full Gleam opened in visual studio.

Right now I'd be really happy to see an example SConstruct file which produced a useable project file, where useable means having the ability to modify, build AND debug code in the Visual Studio environment. I'm not an expert in scons or python but it looks to me that we'll need to make our own modifications to scons to get this ability.

I'm happy to keep plugging away to see if this could be a worth substitute for cmt, especially since it has so many positive features, but I have to say that right now I think we would be better served by upgrading what we have.

Tracy 

Followup from Toby Burnett on 3/7/06:

 Yes, I completely agree. I would never consent to a situation where we did not have a solution with all of the packages needed to build a given package. I depend on that. That means that we really need that new bit of code that finds, or constructs all the relevant project files and includes them in the .sln file.

I don't understand your second point. What do you mean by "usable"? It looks like it would satisfy that to me, given the ability to modify the Sconstruct file on the fly. Your other comment about debug symbols I guess means that the default compile flags need to be modified, strange
that "debug" would not include them from the start.

Finally, what do you mean by "upgrading what we have"? What features are we lacking? If it is the conversion to Visual Studio 2005, lets do it! I have a version, but have not yet installed in on glast-ts.

--Toby

Followup from Tracy Usher on 3/8/06:

My second point was that as of last night I was unable to debug within the visual studio environment with solution/project files created by scons MSVSProject. And I cannot compile individual cxx files from within visual studio, am only able to build the entire project (and that, apparantly, only in release mode). I'd offer that means it is not really something that we can use to support the software effort at present. I've not tried to beat down the problems yet today, but I do see how we can at least build packages with debug options. I have also tried to go through the scons e-mail archives to look for other experiences with MSVSProject, so far without a lot of useful feedback (mostly because I can't get their search to work so have to browse by subject).

On the last point, I'm slightly confused I guess. I thought the entire reason scons was even under consideration was that cmt was seen to have problems which needed fixing? From my narrow windows-centric perspective, it seems to me that we only need do the upgrade cmt to windows 2005, and the timescale for doing that is really set by the expected availability of underlying external libraries which need to be built against the new windows libraries (so, don't see a big rush for this). If there are no other issues with cmt then I'm ready to maintain status quo since I'm not seeing any advantages to scons (again, from my windows-centric viewpoint).

Finally, with respect to GlastPolicy, etc. The cleanup that was done in the last round (by Toby and Traudl?) resulted in a much more understandable set of files which closely enough mirror the LHC experiments files that only a couple of small tweaks were needed to build Gaudi on windows. As we continue to use LHC software, there is an advantage in maintaining this connection.

Tracy

Followup from Toby Burnett on 3/8/06:

 Well, let's assume that we can set debug options for building files. But then I too frequently compile single files, the best way to get quick turnaround when fixing compilation errors, and if the project files do not support that, it is indeed a show-stopper.

But I suspect that given the project file flexibility, we could fix that, if we are serious about pursuing this path.

--Toby
 

Followup by Tracy Usher on 3/8/06:

I found these nice examples of augmenting the builds:

http://dev.openwengo.com/trac/openwengo/trac.cgi/browser/softphone-classic/trunk/wengoscons/wengoscons/common

Unfortunately, it still makes the same "style" of project file that we are getting right now.

Tracy

Followup from Navid Golpayegani on 3/8/06:

I'm confused. Isn't scons claiming that they use md5s to make sure  only files that actually have changed are re-compiled? So if I  understand it correctly scons would only rebuild a .cxx file if it's  md5 differs from the last time it built it. Making it unnecessary to  tell Visual studio to build single files.

In fact it seems to go beyond that. It keeps track of the contents of  object files apparently too. So that when you rebuild a cxx it checks  if the object file has changed too. If it hasn't, it prevents it from  rebuilding the library/program it depended on.

The example they give shows that when you edit the .cxx and only  insert new comments (ie no code change), the .cxx is compiled into  a .o and since the .o is identical to the previous version of the .o,  changes are not propagated to other libs/programs that used that .o

Navid

Followup from Tracy Usher on 3/8/06:

Of course, it also then relinks, etc.

And, one nice thing about compiling in visual studio is that when you get a compile error you can click on the error and it takes you to the offending line of code. Often very convenient.

etc.

My concerns are strictly from the development/debug side working within the Visual Studio environment. While building by macro might have some of its own nice features they appear to come at the expense of some useful Visual Studio features and as a DEVELOPER/DEBUGGER I'd hate to give those up. Its why people like me don't debug on linux...

Anyway, I'm still the one confused. Is cmt broken and in need of fixing or not?

Followup from Jim Chiang on 3/8/06:

 Yes, absolutely. This is precisely what scons does, both for compilation and linking, and I can attest to this from first-hand
experience.
 

Followup from Toby Burnett on 3/8/06:

 I think that what Tracy and I appreciate, but Navid and Jim have not experienced, is how blazingly fast is to get feedback about compilation errors for a given file in interactive mode. We are talking about speed, not basic functionality. Surely it is much faster when you are asking to just compile the file that you have open in the editor; it does not then
have to make a detailed analysis of the sort described. Tracy brings up the Visual Studio capability to jump to the line involved in a compilation error. I too absolutely love this. But I'm less concerned about losing it, since it only involves parsing the file and line numbers in the compilation output file.But before I can claim to have strong opinions one way or the other, I need some actual experience that I can compare with the CMT-generated project files. That seems fairly remote, given how busy we all are.

--Toby
 

Followup from Jim Chiang on 3/8/06:

 Of course, on linux, with properly written Makefiles, this sort of thing is easy to do, i.e., compiling a single file. You just have to provide the correct target name, which generally is something obvious. Back in the pre-cmt days, when I used to have properly written Makefiles, I did this all the time. So speak just to your own experiences please.

And if the compile command was issued from emacs one gets the automatic line jumping to the compilation error in the source code. Fortunately, this is still available even with cmt's makefiles. Same goes for running gdb under emacs with regard to runtime errors.