Wednesday
Action Items
- After Brian's discussion with Andy H., it sounds like we have an easy path to move to xroot and eliminate the archiver. We need to contact Wilko or Lance to discuss how
Morning Status
Elizabeth and Joe E will be meeting with Rob over the next 3 afternoons for discussions.
Elizabeth, Don, and friends will continue pouring over Steve Tether's Confluence pages to extract a list of software and their locations.
Brian is busily updating repoman. Later today, he could update the Docker container to use the latest repoman and build GR or ST.
Joanne will be looking at updating the existing GH repo to support SCons builds.
Arash has been in touch with Joanne concerning the dev RM DB. The current DB contains 50 GB. Arash has asked if there is anything that can be pruned. Joanne is looking into that.
Tom G and Nicola had a fruitful meeting covering the reprocessing yesterday.
Alex, Joe A, and Giacomo are continuing to work on conda builds and testing.
A new version of repoman has been released and installed at SLAC on RH6 under $GLAST_EXT/repoman
| Code Block | ||
|---|---|---|
| ||
repoman checkout --in-place ScienceTools-scons repoman_test_2 |
Developer Workflow
Long discussion concerning the developer workflow with many ideas bandied about including a monolithic repo (SuperRepo). Below is the white board Brian drew up which outlined development along branches in multiple subpackages (not all of which involved the same developer or purpose), PRs, triggering builds via github hooks, and creating of a new meta package (ST or GR) release.
Brian showed us how we could perform releases via the Jenkins web interface.
After much discussion it appears we agreed to continue down the path we discussed in Feb and earlier this week, where we will have subpackages in GitHub, and we will set up Jenkins to handle builds, reusing bits of RM as needed.
We have agreed that the build log parser logic will be pulled out of RM and added to Jenkins, so that we can continue to pull out the most pertinent parts of the extensive build log which includes output for all packages not just the one subpackage a developer might be interested in.
Alternate workflow proposal: GlastTools SuperRepo
An alternate proposal was discussed and did not receive consensus. In this model all package repositories would be collapsed into 1 git repository containing the union of ScienceTools and GlastRelease packages. The two products would be considered different build targets for the build system. SCons would continue to be responsible for building the requisite packages for each target, and differentiating between external dependency versions. This would greatly simplify the Continuous Integration model for Jenkins.
Developers raised objections that moving to this model would be too disruptive to their current workflow, one based on CVS tags shared between distinct repositories.
GlastRelease
Matt asked who is going to be taking over GlastRelease. That is apparently Tom Stephens. Heather showed Tom the version that L1 is currently using.
We briefly discussed handing of GR externals. Heather will pursue using conda, leveraging off the extensive work already done by Giacomo and Alex. Giacomo showed Heather their repo, and she'll fork and make some additions for GR and we'll go from there.
Tuesday
Action Items from Tuesday
- Brian to update repoman to provide an init step to detect if a user has SSH credentials available for GitHub, will revert to HTTP otherwise.
- Brian may consider making repoman a conda package
- Richard will contact the Computing Center to determine the fate of astore and recommended mitigation plans for data access.
- Steve may contact Andrew May about "store"
- Don Horner will become the new resident archiver and xroot backup expert and work to craft a replacement
- Elizabeth and company will be writing up a list of ISOC software and their locations in Confluence.
...
Alex's update from yesterday's afternoon session with Giacomo and Joe A. concerning conda builds for Mac.
Alex was working on building the external dependencies for the Mac.
Giacomo worked on getting Travis-CI builds working.
Joe A worked on getting the tests and environment variable setup fixed.
Once the fermi-lat GH area is set up, they will point to the new repo and use that so that we are all sharing the same code base.
...
Installed repoman on RHEL6-64 under $GLAST_EXT/repoman/miniconda3 - use by pointing your path to $GLAST_EXT/repoman/miniconda3/bin
| Code Block | ||
|---|---|---|
| ||
pip install --pre -U fermi-repoman |
GitHub access via SSH is the default connection method for repoman, if using SSH, your repoman checkout command must be modified.
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
repoman --remote-base https://github.com/fermi-lat checkout GlastRelease-scons repoman_test |
Repoman checks out tags of subpackages based on the packageList.txt. For development, one would use standard git commands to update to master, create a branch, make changes and push into the repo. Joanne will be using that mechanism to make her changes to repo to allow SCons building.
A new tag must be applied to all of celestialSources.
Joanne reminded use that for container packages like celestialSource and irfs, the SConscript files reside in each of the sub-sub-packages and we may need to modify the RM build mechanism to accommodate that structure.
Brian has added in an "auto-bump" mechanism to implement tagging for version, minor, and patches, similar to what we had in place for stag.
The Steve Tether Show
ISOC software summary - Steve's summary from Feb 2017
Archiver
While in the past it performed many functions, the archiver's remaining take is to move Raw data + FastCopy data to tape via "astore". Due to losing the ability to obtain an AFS token, much of the archiver's functionality has been lost, including the ability to clean up data on wain031 that has already been backed up. Steve now does this manually. The erase occurs when both files "astoredone" and "xrootdone" are created, meaning that both backups have been performed and the age of the raw data is sufficient.
on wain031: Note that there is a replacement disk, and there is a slow copy of this area occurring now.
isoc-flight
---> u23
----> archive
------> L0
------->fastCopy
Steve's opinion is that we should replace the archiver rather than attempt to fix it up.
...
"astore" is deprecated, Computing Center is planning to discontinue its use at some undetermined time in the future. Brian suggests we may have 2 years, but that needs to be confirmed by the Computing Center. Andrew May is the current "astore" maintainer. Richard also mentions that Lance is very helpful.
- Deal with old data
- Provide alternate "astore" capability for future data.
The idea was floated to migrate all the old data, but that would likely require hundreds of hours of I/O.
Richard confirms that we have plenty of tape available.
Steve owns the L0 task.
Rob asks about the DTR task contained in the fastCopy archive. Steve indicates that no changes are required for the DTR task if astore is replaced. "astore" only handles back ups.
CHS package handles unpacked LAT raw data, lsf an ldd formats. This is used heavily by the half-pipe which extracts evt files and passes them to L1.
Issues on some RH6 machines have been noted.
Don Horner was named as the lucky person tasked to learn about the current archiver and craft a replacement.
Monday
Attendees: Joe E, Joe A, Tom S, Regina, Giacomo, Jermey P, Rob C, Tom S, Elizabeth F, Don H, Alex, Joanne, Richard, Tom G, Heather, Brian, Matt W, Eric (afternoon), Jim C
Action Items from Monday
- We will migrate to GitHub without the use of git tools like subrepo as a first step. Brian will introduce a fresh copy of ST and GR.
- Joanne will test introducing the --rpath flag in the SCons builds.
- At Jim's request, Heather will contact Johan about his needs for ST installations and development.
- Giacomo and Alex will pursue using condo-forge for their conda builds.
- Giacomo and Alex will contact the maintainers of the current condo-forge cfitsio channel.
...