Dashboard > SAS: Science Analysis Systems > ... > Infrastructure > WhiteBoard
  SAS: Science Analysis Systems Log In View a printable version of the current page.  
  WhiteBoard
 Browse Space
Added by Heather Arrighi, last edited by Joanne Bogart on Mar 29, 2007  (view change)
Labels: 
(None)

Burning Issues

External Libraries

  • We're falling behind in versions in most of our external libraries. Their sources are disappearing. They are no longer supported on newer platforms (RHEL4, Macs, etc..). Examples:
    • fftw (no longer has source available, doesn't compile on intel mac with gcc4)
    • gaudi (no longer has source available, requires minor source changes to work on rhel4)

Unit test Failures

  • we have a handful of packages in GR and (I think) one in ST whose unit tests that routinely fail or enter infinite loops.
  • perhaps there should be a review of what the tests are testing (if anything)

"Pointer" Skims

  • Eric Charles wrote a scheme to point to events in other files. This could be a good way to handle interleaved events and also take some of the pain out of opening N Root files in lock step. Could also come in handy for listing events to use for calibrations.

Dependencies and our use of CMT (Toby)

  • Our dependencies are out of control: CMT has no way of evaluating whether the many includes for compilation, or explicit librarires for linking, are really needed. This needs a careful review of all our requirements files: are the use statements really needed? Can they be put in the private section? Should they have a "-no_auto_imports", which hides clients from unneeded dependencies? (This may then require "-imports" qualifiers.)
  • Closely related to this is the fact that many such dependencies were generated by the needs of test programs. A very good example of this is RootIO, which exports a slew of dependencies to all its client packages, but has few dependencies to actually build itself.
  • CMT provides a way to hide its complexities: patterns.  One defines a few basic requirements files, and implements them with patterns. This was applied in the Science Tools, but we were always too busy to try to reorganize the Glast Release and Beam Test packages.

Environment variables (Toby)

Our use of CMT leaves a very heavy footprint for executables, which is a big strike against it. We are using it in a default mode in which it forces the definition of two variables for each package, namely <package>ROOT and <package>CONFIG. We never use the latter, but sometimes need the former.  An example is the package Gleam: running the setup to define all the environment variables adds 255 variables!

This is easy to turn off, but any such assumption then needs an alternative environment variable, most often to find the root of the package. Having this explicit in the requirements file makes this public, a plus. This has been done for FluxSvc.

Test Programs (Toby)

We started a project to define test programs as sub-packages, allowing the dependencies to be factored. (A fix for much of the dependency problem mentionsed above.) This was done for FluxSvc, with a simple convention: if a package had a subpackage named "test_<package>", then that defined a test program for the package. Such a package does not show up with a CMT "show packages" command, but it is easy to check for the existence of, say, test_FluxSvc/cmt/requirements, then, when cd'ing to that folder, CMT recognizes the package. This is supported by MRvcmt, and apparently by the RM only on Windows. This ball was dropped, needs to be put back in play.

 MRStudio (Joanne)

 The design of MRStudio (gui layered on the core functionality) means that there can be much more commonality in the way RM does its builds and the way developers do, easing the burden of maintaining separate code to do essentially the same thing and making RM a more useful tool for debugging build problems on "the other" platform.  This project deserves high priority.

Shared Libraries for Utility Packages (Joanne)

Currently facilities and detModel are built as static libraries rather than shared (it can't be a Gaudi-style component shareable of course, but it could be linked shared, similar to xmlBase and other packages).  facilities is used by many (perhaps most) other packages.  We are forgoing the benefits of a shareable (faster linking, less duplication of binaries, easier to propagate facilities bug fixes or improvements to all packages using it) for no reason that I am aware of.  For detModel, used only by GlastSvc and the diagnostic package detCheck, the benefits are not as great, but the principle is the same. This is not a critical need, but it is an annoyance and there is some overhead involved in maintaining two branches of facilities (the main one building a static library for Gleam and ScienceTools; the other building a shareable for use in the Online environment).

Children   Hide Children | View in hierarchy
SCons progress (SAS: Science Analysis Systems)

Comments  (Hide)

(Joanne) About the dependency item and our use of CMT in general:

There is no panacea.  We have an inherently complex collection of code which we need to support on platforms (Windows and gnu-based) with very different building styles. CMT has serious fundamental design flaws (e.g., no distinction between whether a dependency is at compile time, at link time, or at run time, nor whether package A depends on all the products of B or only some of them).  While use of some of the features CMT provides, like patterns and -no_auto_imports, can help in some situations, it can make the resulting requirements files even harder to understand and their behavior harder to predict for most of us. 

We could consider moving to a new system altogether, like SCons, where there might be a better chance of expressing the true dependencies more accurately and explicitly, but first we would need to make a realistic estimate of the costs in person-power and potential disruption. On that basis I can't see it happening between now and launch.  Or we have to make do with CMT.  In that case we shouldn't delude ourselves that we'll ever get to something which is truly satisfactory, but it is still worthwhile to look for areas where an acceptable amount of work will lead to significant improvement (subpackages for test programs might be an example) or a very small amount of work will lead to  a noticeable improvement.

Posted by Joanne Bogart at Mar 29, 2007 17:27

(Jim) Moving forward without discarding CMT:
It seems to me that CMT provides a mechanism for integrating a build tool like SCons (which does not do package management) into our system without causing undo disruption. I can imagine a scheme whereby we use the document mechanism to generate SConstruct files (the equivalent of Makefiles) for each package. The CMT fragments could be written to use the cmt show uses command to ascertain the inter-package dependencies and launch a python script to generate the SConstruct files at the same time the standard CMT Makefiles are generated. Clients would then have the choice to use the old Makefiles or VS solution files for building or use the new SConstruct files instead. The advantages of the SConstruct files is of course that they can be written to be multi-platform (i.e., one SConstruct file that would work on both Windows and Linux) and more importantly, SCons keeps track of inter-package dependencies and thereby provides more reliable incremental builds.

Posted by James Chiang at Mar 30, 2007 08:31

(Joanne) Could you please define "package management"?  I'm having trouble sorting out what CMT would do and what SCons would do.Thanks.

Posted by Joanne Bogart at Mar 30, 2007 18:50

In this context, I take "package management" to mean everything except building the code. Under this proposal, we would still use CMT to organize and checkout packages and use it to establish the package-level dependencies via use statements. Here the SConstruct files would merely be a replacement of the Makefiles currently being used and that's all. Incidently, I erred in stating when the SConstruct files would be generated. As I understand CMT, they would generated during the "make" step via the document mechanism and not during the "cmt config" step, so the Makefiles would still have to be invoked, but rather than using "make" to compile the code, SCons would do it, either indirectly as part of a Makefile fragment or directly by the developer when he or she wants to build the code after the package has already been "config"'d. Unfortunately, doing things this way will not necessarily solve the problem of having every package link against every library that is used, directly or indirectly, by that package, unless we start using things like "-no_auto_import" consistently in the requirements files themselves.

Posted by James Chiang at Apr 02, 2007 14:46

I made two assertions that I would like to repeat:

  • A major restructuring, as opposed to minor tuning, has to be well before launch. After that, no way. So I think that this is the best, or least worst time. The trade off is between potential short term disruption and long term maintenance and stability. Obviously I lean toward the long term.
  • The risk is easy to assess: a new system must demonstrate that it works first, and must work in parallel. This would apply to each stage of a multi-stage transition.

On the second point, I hear from Jim that Navid has already demonstrated some sort of SCons solution for ScienceTools. If the subject is now open for discussion, I would love to hear the details.

Posted by Toby Burnett at Apr 01, 2007 12:29
Powered by Atlassian Confluence, the Enterprise Wiki. (Version: 2.5.8 Build:#814 Oct 02, 2007) - Bug/feature request - Contact Administrators