You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 16 Next »

Tuesday

Tom & Nicola meeting to discuss reprocessing.

Warren, Don, Joe E, Tom S, Elizabeth meeting to discuss calibrations.

Repoman

Installed repoman on RHEL6-64 under $GLAST_EXT/repoman/miniconda3 - use by pointing your path to $GLAST_EXT/repoman/miniconda3/bin

GitHub access via SSH is the default connection method for repoman, if using SSH, your repoman checkout command must be modified.

SSH GitHub access
repoman checkout GlastRelease-scons repoman_test
HTTP GitHub
repoman --remote-base https://github.com/fermi-lat checkout GlastRelease-scons repoman_test

 

Monday

Attendees: Joe E, Joe A, Tom S, Regina, Giacomo, Jermey P, Rob C, Tom S, Elizabeth F, Don H, Alex, Joanne, Richard, Tom G, Heather, Brian, Matt W

Action Items from Monday

  • We will migrate to GitHub without the use of git tools like subrepo as a first step.  Brian will introduce a fresh copy of ST and GR.
  • Joanne will test introducing the --rpath flag in the SCons builds.
  • At Jim's request, Heather will contact Johan about his needs for ST installations and development.
  • Giacomo and Alex will pursue using condo-forge for their conda builds.
  • Giacomo and Alex will contact the maintainers of the current condo-forge cfitsio channel.

 

Topics overview from Richard

Package Manager:  RM vs Repoman and what are our developer methodology for the next 10 years.  Get the biggest bang for the buck for our efforts.

Need to merge the efforts of the FSSC and SLAC.  The conda distributions may be part of the release management process as one of the build products.

May have limited conversations about containers this week.

Mac OS support - how do we manage that?

 

Start mind-meld with Warren about L1 processing.  Rob will be talking to Joe E and Elizabeth about mission planning and more!

Steve T and the plans for the ISOC - he's hoping not to move past RHEL6.  The archiver, written in Perl, is a complete black box.  Troy Porter and company may look into a rewrite.

Review the skimmer and what we are willing (or not willing to do) to support it.  Command-line and web interfaces.  TSkim and L1Processing interfaces.  Might contact David Chamont. 

Tom G and Nicola are our resident reprocessing experts.

Brian presents a StrawMan Release Workflow

         

  • commit/tag 
    • Release  = commit + tag
    • Travis or Jenkins driven 
      • sets ENV vars
  • INIT WS
    • Use reporman to init workspace
      • workspace only works currently for ST and GR due to need for packagelist.  Could use packageLib.py to determine dependencies and requirements to allow per package builds.
    • If including extlibs into containers, every part of this setup would be part of the containers.
  • Code Generation and DOxygen
    • Code Generation (SWIG) but that is done in SCons
    • Really just doxgygen for now
  • Compile
    • In Jenkins or Travis this would do a loop over available systems to build and variants
      • CentOS6, Mac, CentOS7 - this info comes from?  which is currently in RM DB.. Brian would like to put that in the repository, due to a desire to avoid putting those credentials into Travis. Could have additional matrix for the compilers to use.
    • SCons builds the unit tests too
  • TEST
  • Package
    • package env that the build used, including dependencies and externals, which can ultimately be pulled into a container
    • SCOns currently create the tarball and the setup scripts
  • Validate
    • We don't currently have any system tests for ST, we do have some for GR which are not currently run as part of RM, it would be nice to reactivate for this for releases.
  • Deploy
    • Moving tar balls from workspace into NFS to CVMFS
    • Containers (Singularity) will link to CVMFS and includes xrootd installed
    • Could deploy to conda too

Giacomo reminds that the conda build system includes most of the above.  Are we going to consider moving everything to the conda system or in parallel?

Richard:  who are the recipients of the builds? processing, developers, collar users..

conda - is agnostic to what you use for the build (hmake, scons, etc)

Giacomo mentions that conda builds requires some things like the to use rpath, need to compile,   Joanne volunteers to try to use rpath for a build.

Brian:  conda could be another variant in the COMPILE box.

Brian: Release process creates the tags, by the user or via something like Jenkins.. provide package name and the tagged version that you want.

unit test - results

build logs - Jenkins stores the output.  Those familiar with the RM wonder about extracting the errors for developers to more easily view build problems, as we already do for the RMII web pages.

Matt: Are we completely replacing the RM?  
Brian: replacement of RM ultimately, but re-use RM for the beginning phases.

Joanne:  RM is tag driven

Alex: Travis is commit driven
$130 for the unlimited time frame for an indeterminate period of time
Modularization will also help.

Brian: WS is semi-persistent - if doing a repoman checkout, if only one sub package changes, then that's the only one checked out and SCons is smart enough to only build what is necessary.

 

Monday Afternoon

Discussion about git tools such a subrepo, submodule, etc occurred during the morning (Someone could insert their recollections).  We discussed it further in the afternoon, where Joanne mentioned that subrepo handles some operations nicely, particularly when branching is utilized.  We ultimately agreed that subrepo could be added later if we find it useful.  That way, setting up GitHub could proceed immediately.  Next step will be to use repoman to checkout the code, test a build using SCons and then feed the RM to start up a build.

Brian has recently gone through the ST and GR dependencies and created 65-70 repos.  This includes some redundant copies of some repos for package that are shared between GR and ST.  Will push the repos but notes that he needs to prune large files.  

Conda Discussion with Giacomo and Alex

https://github.com/giacomov/conda-fermi-externals/blob/master/notes/How%20conda%20works.ipynb

Giacomo went over the finer points of setting up conda builds for the externals.  See the detailed notes above. Matt asked about using condo-force instead which may be a better long term solution as they handle the builds for us.  It was noted that condo-forge may be using a later version of gcc and there is a cfitsio channel already in place. Matt suggests contacting the cfitsio developers and trying out condo-forge.

Testing the conda binaries of ST is a work in process, with many issues revolving around the initialization script that sets up the environment.

Discussed potential time limit issues with building ST via Travis-CI.  Without externals, Joe A estimates it takes 30 minutes which is under the Travis time limit (~40min?)

Mention of Travis led to a discussion of Mac OS support.  Questions swirled about how to support Mac developers. Eric notes that he has all but given up on Mac support. Jeremy wondered if one could use the condo-build mechanism to support development on the Mac.  It sounds like the build.sh could be reused locally, with tweaks to remove the installation step. Matt suggests introducing additional compiler versions in the CI may help identify many of the problems discovered in the Mac builds by the FSSC before releases are made by the LAT. Jim suggests adding clang as one of the options in the CI.

Alex and Giacomo ran off to work on support Mac in the condo-builds.

 

 

  • No labels