Goals
The immediate goal to is put source for ScienceTools_User (somewhat reduced, non-ROOT version of the complete ScienceTools now built at SLAC) in GitHub as well as continuing to maintain it in CVS. Developers would work from GitHub. Changes there would be automatically reflected to CVS, allowing Release Manager to continue to be triggered by new tags in CVS. Subsequently we might want to
- Keep the larger distribution of ScienceTools in GitHub as well
- Keep GlastRelease in GitHub
- Modify Release Manager to work off of GitHub (or use something else altogether for builds, testing and distribution)
The first hurdle is our relatively complex package organization.
Background
Subpackage structure and sharing
For purposes of this discussion ScienceTools_User (ST_U), ScienceTools (ST) and GlastRelease (GR) will be known as containers; the taggable components will be called subpackages. Apart from the special subpackage containerSettings ST_U is a proper subset of ST. Certain subpackages of ST and ST_U also belong to GR. ST_U doesn't officially exist yet, but all the issues discussed would apply to ST_U as well as ST.
Physical structure
Much of the organization and all the sharing is handled in the CVS repository by means of sym links. There are top-level directories ScienceTools-scons and GlastRelease-scons. There are also top-level directories for subpackages such as astro, Likelihood, etc. In almost all cases the subpackage directories are sym-linked into the container directory. Shared subpackages like astro and celestialSources are sym-linked into both. The one exception is containerSettings. Each container has its own version of this subpackage, which is a regular subdirectory of the container directory. When you check a container out of CVS, sym links are followed, so you end up with all source files in regular subdirectories.
Subsubpackages
There are two subpackages (irfs, celestialSources) which have subsubpackages (that is, separately taggable physical subdirectories). celestialSources belongs to both ST and GR. SCons and Release Manager recognize subpackages and subsubpackages by the existence of a file called SConscript. In addition to specifying to SCons how to build the package, this file also contains version information.
Handling subpackages and subsubpackages
The single fact causing us the most grief may be this: CVS tags apply to a directory (optionally recursively); git supports tags only of entire repositories. Hence we surely need separate repositories for subpackages and a way to reassemble the hierarchy when they're checked out. It's less clear what to do about subsubpackages. There are only two subpackages with subsubpackages: celestialSources and irfs. Our custom subpackage-tagging software goes to extra trouble to avoid tagging subsubpackages when the parent subpackage is tagged.
Strategy summary
- Put subsubpackages in their own repositories in Git; use some existing tool, like subtrees or subrepos, to keep track of the "real" structure
- Put subsubpackages in their own repositories in Git; use homegrown scripting to bridge the gap between git structure and CVS structure
Eliminate subsubpackages by changing organization in CVS and updating references as needed
Git or third-party "Sub" tools
From the documentation for subtree and subrepo it seems likely that git submodule, the original and fully-integrated-into-git attempt to solve this kind of problem, is not suitable for our case (three levels of repositories, none of them external and all potentially undergoing active development). The other two, at least, deserve a closer look, but the discussions in the documentation cited above are sobering. Before trying to decide if any of these are suitable for us we should think hard about how we plan to do development. Using clones? Branches? Or just commit to master, identifying known good commits with tags and avoiding conflicts among developers with informal communication?
Scripting bridge
Each subpackage or subsubpackage (that is, anything with an SConscript in its top directory) goes in a separate repository, as with previous strategy, but we write our own custom tool to help with the scenarios we expect to encounter, and not much more.
Flatten hierarchy
Having a third level to our hierarchy introduces significant complication. Since there are only two packages where this is an issue it's tempting to change the structure to make the subsubpackages full fledged subpackages. This would affect other subpackages somewhat, e.g. in #include references; we need to make a survey to find out just how much work this would entail. In the case of celestialSources, this would impact GR as well as ST and ST_U.