...
The current structure of these containers is not suitable for GitHub because certain files (SConstruct, allExternals.scons, ..) are shared by all containers via sym links into their root directories. These files need to be moved into a subpackage which can be treated like other subpackages. Such a scheme is under development with the new subpackage SConsShared.
Syncing repositories
Because of the container reorganization needed to use SConsShared I don't believe there is any hope of moving development for all containers to GitHub initially; it will have to be done piecemeal. Neither will modifying ReleaseManager (or substitute) to work from GitHub happen instantaneously. That means we will have to live with the "same" source in both GitHub and CVS. We need reliable syncing mechanisms for code and tags.
(Aug. 29, 2017) UPDATE: an informal poll reveals that developers are willing to tolerate an outage of weeks in order to get this done "all at once". This means we don't have to put effort into figuring out how to reliably keep git and CVS in sync.
Physical structure
Much of the organization and all the sharing is handled in the CVS repository by means of sym links. There are top-level directories ScienceTools-scons and GlastRelease-scons. There are also top-level directories for subpackages such as astro, Likelihood, etc. In almost all cases the subpackage directories are sym-linked into the container directory. Shared subpackages like astro and celestialSources are sym-linked into both. The one exception is containerSettings. Each container has its own version of this subpackage, which is a regular subdirectory of the container directory. When you check a container out of CVS, sym links are followed, so you end up with all source files in regular subdirectories.
...
- Put subsubpackages in their own repositories in Git; use some existing tool, like subtrees or subrepos, to keep track of the "real" structure
- Put subsubpackages in their own repositories in Git; use homegrown scripting to bridge the gap between git structure and CVS structure
Eliminate subsubpackages by changing organization in CVS and updating references as needed (but we still need an existing or custom homegrown tool which handles the two levels)
Git or third-party "Sub" tools
- Git submodule doc
- Git subtree doc (May, 2015) more doc by the same guy (May, 2013; updated Jan. 2017)
- Git subrepo doc, source
From the documentation for subtree and subrepo it seems likely that git submodule, the original and fully-integrated-into-git attempt to solve this kind of problem, is not suitable for our case (three levels of repositories, none of them external and all potentially undergoing active development). The other two, at least, deserve a closer look, but the discussions in the documentation cited above are sobering. Before trying to decide if any of these are suitable for us we should think hard about how we plan to do development.
...
In CVS we've reserved branches for large-scale differences, usually only suitable for GR and have mostly followed model 3), but branches are easier to use in git. 1. seems like overkill for the small number of developers involved.
Both subtree and subrepo appear to handle branches reasonably well. In the more recent subtree doc the description of how to push changes to the subtree back to the remote is a bit daunting, especially since this is likely to be a common operation for developers. subrepo doesn't have this issue, in fact fixing it was one - perhaps the chief - motivation for writing it.
I don't see any discussion of tags in the documentation of any of these tools but, since we've traditionally used home-grown tools rather than naked CVS commands for tagging already (C++ program stag for individual subpackages; tagCollector.py for release and release candidate tags), perhaps this isn't so important.
My tentative conclusion is that we can eliminate submodule and subtree from consideration; subrepo should be investigated further. In particular we should see how it does with a three-level hierarchy.
See also this discussion of reorganization and beginning experiments with subrepro.
Scripting bridge
Each subpackage or subsubpackage (that is, anything with an SConscript in its top directory) goes in a separate repository, as with previous strategy, but we write our own custom tool to help with the scenarios we expect to encounter, and not much more.
...