In this page we will document how to create a new release.

Ana vs Dm

There are two kinds of releases

  • dm - data management
    • build six environments with each release - py27 in dev/prod for rhel5/6/7 
  • ana - analysis, this is the one that is user facing
    • build 13 enviornments with each release, py27 and py3 in dev/prod rhel5/6/7, and a gpu in dev/rhel7
    • build a new psana_conda package for each release
    • test the releases

Overview

If we are going to build version n.n.n, then basically just

  1. sudo as psreldev on psel701
  2. For ana (not dm) prepare to build new psana_conda during automation:
    1. run kinit davidsch (or whoever you are)
    2. bump up the version number for psana-conda in its recipe's meta.yaml (in /reg/g/psdm/sw/conda/manage/recipes/psana/psana-conda-opt/meta.yaml)
    3. bump up the version for psana-conda to the same in the anarel.yaml (in /reg/g/psdm/sw/conda/manage/config/anarel.yaml)
  3. Check that conda packages we maintain are up to date (logbookclient, ndarray, openmpi, etc)

    1. Run this command on each of the rhel5/6/7 environments:

      ana-rel-admin --cmd bld-all

        ana-rel-admin --cmd bld-all (which runs relatively quickly) does the following

      1. reads this file: manage/config/pkg_build_order.yaml
      2. builds conda packages from all listed recipes
      3. run this on each of the rhel5/6/7 environments.
      4. note build order, if recipe B requires package from recipe A, list them a A than B  
  4. To actually perform a build for ana:  (NOTE: make sure $USERNAME is set to psreldev otherwise you can get a paramiko authentication failure.  If you "sudo su psreldev" USERNAME can be set to "root")
    1. Run ana-rel-admin --cmd auto --ana --name n.n.n
      where n.n.n is this version number
  5. Similarly, for dm
    1. Run ana-rel-admin --cmd auto --dm --name n.n.n 
  6. tag management code
  7. change ana-current (or dm-current)

Below we go over more detail for general admin.

Get to build node and admin account

When you are sudo-ing to use the admin account, you may want to do graphics.

To do this, you have to get the ssh key for yourself and share it with the admin account.

Below includes some technical instructions for doing this (thanks to Jason Koglin).

Note, if you are content with a terminal editor like vi, or emacs -nw (no window) then you can skip those commands.

ssh -Y pslogin
ssh -Y psel701
# I have been doing all admin work on psel701, but any 
# rhel7 machine with internet access should work
 
# skip if you are a terminal editor
echo $DSIPLAY
 
# say it is 11
xauth list | grep unix:11 | grep psel701
 
# say I see
psel701.slac.stanford.edu/unix:11  MIT-MAGIC-COOKIE-1  ef89ef89ef87feaab871203928cb918d
# copy that line to the clipboard for future reference
 
sudo -i -u psreldev
 
# skip if termimal editor person
xauth add psel701.slac.stanford.edu/unix:11  MIT-MAGIC-COOKIE-1  ef89ef89ef87feaab871203928cb918d

At this point, you are psreldev. You may want to note customizations done in .bashrc.

Right now, I do

export CDIR=/reg/g/psdm/sw/conda
source conda_setup --dev --env manage

So you will be in the manage environment for the appropriate rhel5/6/7 development installation.

CDIR is not used by the system, just for my convenience.

Update Ana-rel-admin Code

The management code is in $CDIR/manage.

It is under version control on github: https://github.com/slaclab/anarel-manage

Between releases, we may make little edits or add packages we want in the next release. Take a look at the repo history: https://github.com/slaclab/anarel-manage/commits/master

Check the github repo issues, anything we should work on before building a new release?

do a

git fetch
git diff master origin/master

to take a look at any changes. If you like them, merge them to get the management code up to date with what's on github.

For instance, it is common for us to just go to github and edit the anarel.yaml to add a package. We might make a mistake with this which will come up during the build.

Should we update conda and conda-build? This requires special attention. Let's leave this issue open: https://github.com/slaclab/anarel-manage/issues/17 and update it when trying to update conda and conda-build.

Update Locally Built Packages

Normally, one does not need to do this, but do we need to update the version of any of these packages?

  • openmpi
  • hdf5
  • h5py
  • tables
  • keras
  • tensorflow

This requires some special attention. Packages need to be build on all 3 hosts, and the hosts all have to have internet access. We use

  • rhel5 - psdev106
  • rhel6 - psdev105
  • rhel7 - psel701

Note that ana-rel-admin has the command

ana-rel-admin --cmd bld-all

That reads the file config/pkg_build_order.yaml to know the order to build the packages. You can run this in parallel on the three hosts.

When re-building a package, take a look at the recipe. You may need to get a new source file from the internet and put it in $CDIR/downloads/otherpkgs

Update Psana-conda Recipe

Presently, the ana version is kept in lock step with the psana-conda version. Note:

  • ana-n.n.n  the name of a conda environment that contains version n.n.n of the conda package psana-conda
  • psana-conda the name of our conda package that contains psana

Suppose we are currently on ana-1.2.0 and we want to build ana-1.2.1. What you need to do, is, as psreldev, two things

  1. go to $CDIR/manage/recipes/psana/psana-conda-opt
    1. edit meta.yaml
    2. change the version from 1.2.0 to 1.2.1, for example, this line of the meta.yaml
  2. go to $CDIR/manage/config
    1. edit anarel.yaml
    2. find the line with psana-conda, for example, this line of anarel.yaml
    3. change the version from '=1.2.0' to '=1.2.1', not the quotes to specify a string.

 

Automatic Build

Now run, from only the psrel701 host, first do

kinit davidsch

or however you are, this is so psreldev can check things out of svn. Next do

ana-rel-admin --cmd auto --ana --name 1.2.1

If you are making a release 1.2.1. The script will check steps 1 and 2 above. It will then verify that it should build ana-1.2.1:

########## AutoReleaseBuilder ############
psana pkg name: psana-conda-1.2.1
ana release name: ana-1.2.1

is this Ok? based on --name: y

Hit y to continue. Next you might get some output like

--- AUTO: log directory /reg/neh/home/psreldev/public_html/builds/auto-1.2.1 exists, picking up where last AUTO failed
--- AUTO: deleting old file: /reg/neh/home/psreldev/public_html/builds/auto-1.2.1/index.html
--- AUTO ----------
enter password for test_user: davidsch

this is not echoed to screen, but is stored in python string.

The build will test the conda environments from a account other than the account used to build the environments. It figures out who is sudo-ing as psreldev and asks for their password.

You will be able to view the progress of the build at

https://pswww.slac.stanford.edu/user/psreldev/builds/auto-1.2.1/

but note this site gets overwritten if the build fails and is redone. It is common to see the build fail, for instance you forget to do kinit. Just enter the same command. Any steps that were already finished will be repeated.

If you want to start the build over from the very beginning, you can do

ana-rel-admin --cmd auto --name 1.2.1 --clean --ana (or --dm)

that is add the --clean flag. If you are late in the build and need to overwrite previously built environments or other artifcats of the build, you can do

ana-rel-admin --cmd auto --name 1.2.1 --clean --force --ana (or --dm)

but BE CAREFUL WITH FORCE. If you have the name wrong, i.e, you are building --name 1.2.0 and we already have a 1.2.0, you will delete the old environments.

Fix Testing Issues

auto runs tests from the anarel-test package that is checked out in $CDIR/anarel-test. It tests for the presence of many python imports, binaries, and shared object libraries. Newer versions of packages may deprecate one of these, breaking the test. Update the test code.  

Update ana-current

After a successful build, first take a look at the log report, i.e, the https://pswww.slac.stanford.edu/user/psreldev/builds/auto-1.2.1/ for your build. Make sure there are no surprises about packages being dropped or changed. You can always activate the new environment yourself and do more testing.

Then it is time to update ana-current. Enter the command

ana-rel-admin --cmd change-current --ana --name ana-1.2.1

this is interactive. the tool will go through the three envionments we maintain, the py27 (with psana) the py3 and gpu environments. You will get questions like:

Changing ana-current, variant=standard

current content of /reg/g/psdm/sw/conda/ana-current/ana-current:
ana-1.2.0

==== Which installations have environment: ana-1.2.1 ====
prod-rhel5: No
prod-rhel6: No
prod-rhel7: No
dev-rhel5: No
dev-rhel6: No
dev-rhel7: No
 
====
Given report on  ana-1.2.1 - switch ana-current? (y/Y)

Here, you would say no. Before switching to ana-1.2.1, that environment needs to be present in ALL SIX miniconda installations. Likewise for the py3 environment, however the gpu environment is only built in the dev-rhel7 installation.

Tag Management Code

If you just build ana-1.2.1, create a ana-1.2.1 tag for both the anarel-manage and anarel-test packages.

Update Release Notes

That is the page Conda Release Notes. There should be a ana-next entry there. Change that to ana-1.2.1. Copy the little bit from ana-1.2.0 that gives you links to the build log (and edit the links to go from 1.2.0 to 1.2.1). Add any additional notes of importance.

 


  • No labels