...
The first hurdle is our relatively complex package organization. Even a two-level hierarchy (container + subpackages) requires some sort of glue to get the right results for operations like tagging and cloning; the three levels we now have make the job harder.
Background
Subpackage structure and sharing
For purposes of this discussion ScienceTools_User (ST_U), ScienceTools (ST) and GlastRelease (GR) will be known as containers; the taggable components will be called subpackages. Apart from the special subpackage containerSettings ST_U is a proper subset of ST. Certain subpackages of ST and ST_U also belong to GR. ST_U doesn't officially exist yet, but all the issues discussed would apply to ST_U as well as ST.
The current structure of these containers is not suitable for GitHub because certain files (SConstruct, allExternals.scons, ..) are shared by all containers via sym links into their root directories. These files need to be moved into a subpackage which can be treated like other subpackages. Such a scheme is under development with the new subpackage SConsShared.
Syncing repositories
Because of the container reorganization needed to use SConsShared I don't believe there is any hope of moving development for all containers to GitHub initially; it will have to be done piecemeal. Neither will modifying ReleaseManager (or substitute) to work from GitHub happen instantaneously. That means we will have to live with the "same" source in both GitHub and CVS. We need reliable syncing mechanisms for code and tags.
Physical structure
Much of the organization and all the sharing is handled in the CVS repository by means of sym links. There are top-level directories ScienceTools-scons and GlastRelease-scons. There are also top-level directories for subpackages such as astro, Likelihood, etc. In almost all cases the subpackage directories are sym-linked into the container directory. Shared subpackages like astro and celestialSources are sym-linked into both. The one exception is containerSettings. Each container has its own version of this subpackage, which is a regular subdirectory of the container directory. When you check a container out of CVS, sym links are followed, so you end up with all source files in regular subdirectories.
...
From the documentation for subtree and subrepo it seems likely that git submodule, the original and fully-integrated-into-git attempt to solve this kind of problem, is not suitable for our case (three levels of repositories, none of them external and all potentially undergoing active development). The other two, at least, deserve a closer look, but the discussions in the documentation cited above are sobering. Before trying to decide if any of these are suitable for us we should think hard about how we plan to do development.
- Using
...
- forks?
- Branches?
- Or just commit to master, identifying known good commits with tags and avoiding conflicts among developers with informal communication?
In CVS we've reserved branches for large-scale differences, usually only suitable for GR and have mostly followed model 3), but branches are easier to use in git. 1. seems like overkill for the small number of developers involved.
Scripting bridge
Each subpackage or subsubpackage (that is, anything with an SConscript in its top directory) goes in a separate repository, as with previous strategy, but we write our own custom tool to help with the scenarios we expect to encounter, and not much more.
...
Having a third level to our hierarchy introduces significant complication. Since there are only two packages where this is an issue it's tempting to change the structure to make the subsubpackages full fledged subpackages. This would affect other subpackages somewhat, e.g. in #include references; we need to make a survey to find out just how much work this would entail. In the case of celestialSources, this would impact GR as well as ST and ST_U.
Typical operations
The optimal choice of file organization and tools depends on what people and automated systems for building, testing and creating release need to do. It should not be hard to take care of users, who only need to run the code, no matter what choices we make, so here "people" really means "developers".
Developers
- Clone container and all contents from git into working directory
- Start with distribution of tagged release (including source) in working directory; clone individuals subpackages to replace distributed version in working directory.
- Push changes to GitHub for purpose of backup, sharing with other developers (e.g. on a branch?)
- New code ready for production (e.g. push or merge into master branch)
- Tag a subpackage
- Tag container and all subpackages (and subsubpackages) with a release candidate tag (e.g. ScienceTools-HEAD-1-2345)
- Tag container and all subpackages (and subsubpackages) with a release tag (e.g. ScienceTools-11-4-5)