You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

RM, the Narrow View

This section concerns only the code under grits-cpp/ReleaseManager and the databases it talks to.

Application-level functions used by Fermi

Automated functions

  • Automatically make ("HEAD") builds for configured OSes when a new release candidate tag is created
  • Automatically make ("Release") builds for configured OSes when a new release tag is created
  • Automatically make ("LATEST") builds for configured OSes when a new subpackage tag is created

Manual functions

The functions above invoke various executables to accomplish their tasks. These executables may be invoked by hand if, for example, some part of the build output needs to be generated or regenerated.  The RM code also includes some additional executables which may be invoked manually.  Typical tasks which one might want to do manually include

  • Make a build for a particular os-container-variant combination which is defined in the db but not automatically triggered
  • Delete an old build, typically with intention of retriggering

Archiver

A separate application, the Archiver, makes use of the lower levels (lsf, workflow engine).  Under normal circumstances it is invoked periodically via trscron and runs an ancient Perl script. Additional scripts may be run by hand to, e.g. archive or restore a particular file.

RM Structure

RM (excluding Archiver) accesses three MySQL databases: rd_lsf, rd_workflow and rd_releasemgr. 

  • rd_lsf is used for interactions with the lsf batch system. 
  • rd_workflow is used to properly sequence jobs needed for a particular operation.  For Jenkins workflows there is just one entry describing the launch of a Jenkins job; the details (first checkout code, then build, etc.) are handled elsewhere.  For OSes not directed to Jenkins, there are entries in the workflow db describing what has to happen and what steps any given step depends upon.
  • rd_releaseMgr is by far the most complex of the three.  It contains all the information needed to configure builds by os, container (ST, GR, etc.), build type (Release, Release candidate, LATEST) and variant (Debug, Opt) and also information about the status of particular builds (by os, container, build type and tag version).

For each of these databases there is a part of the RM code which talks to it.  The lsf and workflow parts have been untouched for 5 years and even that was a very minor change.  They haven't seen any significant activity for 7 or 8 years. The lsf and workflow parts of the code only need to run where RM itself is running, currently rhel6.  The rest must run on any platform where builds are supported (currently rhel6 and rhel7).

There is considerably more, and more complicated, code talking directly to rd_releaseMgr.  There have been maintenance updates (not exactly frequent, but steady at a low level) right along. Speaking very roughly, the function of this code is, based on data in rd_releaseMgr and on information (about package structure and tags) gleaned from the code repository,  to decide what builds are necessary to create (or delete).  It then does the necessary operations, including updating the database and, in the case of LATEST builds, creating tags in the code repository.

If we used Jenkins for all builds we could perhaps vastly simplify the workflow section of the code.  If we also stopped using lsf maybe we could replace both the workflow and lsf code with something much more streamlined.  For the rest, I don't see any clear path to something significantly simpler.

The RM Ecosystem

In addition to the core functions described above, there are several others which depend to some degree on RM being constituted more or less as it currently is.  For the most part this boils down to a dependence on the rd_releaseMgr database and tagging conventions.

 

 

 

 

  • No labels