You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 27 Next »

LCLS1

Information from Valerio on Jan. 27, 2021.

make sure the psana1-py2 env is used (needed by py2 ana-rel-admin)
cd /cds/sw/ds/ana/conda1/manage/
ana-rel-admin --force --cmd psana-conda-src --name 2.0.6 --basedir `pwd`

cp .tar.gz to where GitHub can see it:
/reg/g/psdm/web/swdoc/tutorials/

psreldev can't write to the above, do as user "cpo" or "valmar"

checkout github feedstock repository:
many feedstocks (all in slac-lcls) e.g.
https://github.com/slac-lcls/psana1-feedstock

update version and checksum in

https://github.com/slac-lcls/psana1-feedstock/blob/main/recipe/meta.yaml

generate new checksum with: sha256sum .tar.gz

sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.10.tar.gz

git push launches the build on GitHub (using GitHub Actions) when it finishes it uploads it
to anaconda.org/lcls-i

this is our real control of the build process (e.g. the GitHub Actions Workflow yaml file gets
generated automatically)

https://github.com/slac-lcls/psana1-feedstock/blob/main/conda-forge.yml

if the above file is modified have to run "conda smithy rerender -c
auto" "-c auto" means automatically make a git commit. but still have
to push to git.

There are a few other cases in which the feedstock needs to be rerendered.
See the conda-forge documentation:

https://conda-forge.org/docs/maintainer/updating_pkgs.html#when-to-rerender

If no relevant changes have taken place, the command will simply not create a
commit. so maybe it is a good practice to run it every time a feedstock is
updated.

"targets" first entry: the conda channel to upload to, second entry is
called a "label" (e.g. a devel and a stable). conda forge always looks
for "main" label.

then "conda create" as usual (as seen in the jenkins CI).

for testing the feedstock, can run the run_docker_build.sh script locally
but requires docker to be installed. The file  "build_locally.py" included in
the feedstock takes care of running run_docker_build.sh

Valerio uses this to test the feedstock on his laptop.

If any dependency is updated (e.g. boost) then feedstocks are rebuilt
automatically, but ONLY if our packages are in conda-forge, which they
are not. To emulate this, Valerio bumps all the feedstock build
numbers for each release (even ndarray because it depends on boost) so
they will get built against the latest versions in conda-forge.

separate feedstock for py27 since everything is pinned in the recipe:
https://github.com/slac-lcls/psana1-py2-feedstock
recipe/conda_build_config.yaml overrides the equivalent in the conda forge
pinning file (most recent version pulled automatically from conda-forge)

in recipe/conda_build_config.yaml (ONLY for psana1-py2-feedstock),
zip_keys: override specification for the 3 special packages python,
python_impl, numpy. conda-forge people helped specify these, in
particular Billy Poon.

feedstocks:
libpressio-feedstock psana1-py2-feedstock sz-feedstock
ndarray-psana-feedstock psgeom-feedstock ztcav-py2-feedstock xtcav2-feedstock
ndarray-psana-py2-feedstock psocake-py2-feedstock psocake-feedstock
psana1-feedstock stdcompat-feedstock

libpressio,sz,stdcompat are all needed by chuck/sz.

update build numbers for all the above for every psana1 release.

Checklist for building psana1 environments from feedstocks

As psreldev on psbuld-rhel7:

  • source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
  • conda activate conda_build
  • cd /cds/sw/ds/ana/conda1/manage/
  • bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.11 --basedir `pwd`

As normal user:

  • cd /cds/sw/ds/ana/conda1/manage/downloads/anarel
  • cp psana-conda-4.0.11.tar.gz /reg/g/psdm/web/swdoc/tutorials
  • sha256sum psana-conda-4.0.11.tar.gz <copy the checksum>

Activate an environment that contains the 'conda smithy' package. For example:

conda activate /cds/home/v/valmar/.conda/envs/conda_forge_build3.9

Then for each of these feedstock (more or less in this order):

            github.com/slac-lcls/sz-feedstock
            github.com/slac-lcls/stdcompat-feedstock
            github.com/slac-lcls/ndarray-psana-feedstock
            github.com/slac-lcls/ndarray-psana-py2-feedstock
            github.com/slac-lcls/libpressio-feedstock
            github.com/slac-lcls/xtcav2-feedstock
            github.com/slac-lcls/psgeom-feedstock
            github.com/slac-lcls/psocake-feedstock
            github.com/slac-lcls/psana1-py2-feedstock           
            github.com/slac-lcls/psana1-feedstock            

As normal user:

  • Modify recipe/meta.yaml
    • Check if new version of packages has been released 
    • If yes, bump up version, update sha256 checksum and reset build number
    • Otherwise, bump up build number
  • Commit
  • conda smithy rerender -c auto
  • Push

After all packages have finished building on GitHub (Unfortunately, GitHub Actions do not provide a "dashboard" to check if the packages have been built or when they are finished. There are only two ways to check this: 1) Manually check every feedstock repository (Under the "Actions" tab) 2) Wait for either the package to appear on anaconda.org/lcls-i or for an email notifying a failure to arrive):

As psreldev:

  • source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
  • cd /cds/sw/ds/ana/conda1/manage/jenkins/
  • conda env create -n ana-4.0.11 --file ana-env-py2.yaml
  • conda env create -n ana-4.0.11-py3 --file ana-env-py3.yaml

LCLS2

First, tag lcls2 and ami repos.

checkout the 20 feedstock packages (in github.com/slac-lcls/): 

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --clone

After these repos are cloned, you can use this to do commands in all repos:

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --cmd "pwd; git pull --rebase"

Either update version (with new git tag) or update the build number in each meta.yaml.  conda-forge enforces manual maintenance of version numbers and sha256 hashes (no automatic determination from latest GitHub tag).  If version has changed, compute new sha256 from .tar.gz from GitHub with:

sha256sum 1.1.7.tar.gz

For a new version, remember to reset build number to zero.  Commit the changes.  This command allows you to modify build numbers.  It will prompt you for each package, and you can enter "y" (increment build number) "n" (don't increment, default) or "0" (set to 0):

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --incbuildnum

The next step requires an environment where "conda-smithy" is installed (we have put this in psrel's conda_build env).  Then rerender with "conda smithy rerender -c auto" (again using feedstock.py --cmd option).  This last command does its own git-commit for you with the changes.  When must this be run?  Valerio writes: "In general any change in conda forge config (for example, when we want to upload to a different channel, or to switch to a different CI). However, it is also needed to pick up the newest version of the conda-forge pinning file, so I run it every time. In the worst case, it tells me everything is up to date and does not create a commit".

The final "git push" of the above changes must be done carefully because it triggers the GitHub build, and order of the GitHub builds matters because of conda's "build:, host:" section dependencies (run-time dependencies do not affect this order).  We believe pure-python packages can go in the first wave, since they have no complex dependencies in "build:, host:" sections of meta.yaml. These are the waves of builds that can be launched in parallel:

  • libnl, libnl3, roentdek, amityping, prometheus-cpp, libusdusb4, psmon, networkfox, rogue, epix, lcls2_timetool, cameralink-gateway, lcls2-pgp-pcie-apps, xtcdata, ami
  • rmda-core (depends on libnl), psalg (depends on xtcdata)
  • fabric (depends rmda-core), psana (depends on psalg)
  • psdaq (depend on psalg, fabric)

The 4 build waves above can be launched with a command like this.

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --cmd "git push" --wave 1

If a package is built for python build matrix tells it to build for the officially supported conda-forge versions (3.6,3.7,3.8).

If we were in conda-forge officially, they take care of these build-order dependencies (but not the versions/sha256/buildnumber).  They may have a bot that tells you that the version was updated, but we still have to update the version number by hand. We haven't gone the conda-forge route because (1) we don't know how much work it is, and (2) cannot upload source .tar.gz to conda-forge, usually get it from GitHub, but the rogue packages are not public.

Unfortunately, GitHub Actions do not provide a "dashboard" to check if the packages have been built or when they are finished. There are only two ways to check this: 1) Manually check every feedstock repository (Under the "Actions" tab) 2) Wait for either the package to appear on anaconda.org/lcls-ii or for an email notifying a failure to arrive.

To create the environment as psrel, use these commands:

conda env create --name ps-N.Nodd.N --file env_create.yaml (devel env)

conda env create --name ps-N.Neven.N --file prod_create.yaml (prod env)

  • No labels