You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

LCLS1

Information from Valerio on Jan. 27, 2021.

make sure the psana1-py2 env is used (needed by py2 ana-rel-admin)
cd /cds/sw/ds/ana/conda1/manage-feedstocks/
ana-rel-admin --force --cmd psana-conda-src --name 2.0.6 --basedir `pwd`

cp .tar.gz to where travis can see it:
/reg/g/psdm/web/swdoc/tutorials/

psreldev can't write to the above, do as user "cpo" or "valmar"

checkout github feedstock repository:
many feedstocks (all in slac-lcls) e.g.
https://github.com/slac-lcls/psana1-feedstock

update version and checksum in

https://github.com/slac-lcls/psana1-feedstock/blob/main/recipe/meta.yaml

generate new checksum with: sha256sum .tar.gz

sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.10.tar.gz

git push launches the build on travis when it finishes it uploads it
to anaconda.org/valmar (currently) but need to switch to it to official

this is our real control of the build process (e.g. .travis.yml gets
generated automatically)

https://github.com/slac-lcls/psana1-feedstock/blob/main/conda-forge.yml

if the above file is modified have to run "conda smithy rerender -c
auto" "-c auto" means automatically make a git commit. but still have
to push to git.

There are a few other cases in which the feedstock needs to be rerendered.
See the conda-forge documentation:

https://conda-forge.org/docs/maintainer/updating_pkgs.html#when-to-rerender

If no relevant changes have taken place, the command will simply not create a
commit. so maybe it is a good practice to run it every time a feedstock is
updated.

"targets" first entry: the conda channel to upload to, second entry is
called a "label" (e.g. a devel and a stable). conda forge always looks
for "main" label.

then "conda create" as usual (as seen in the jenkins CI).

for testing the feedstock, can run the run_docker_build.sh script locally
but requires docker to be installed. The file  "build_locally.py" included in
the feedstock takes care of running run_docker_build.sh

Valerio uses this to test the feedstock on his laptop.

If any dependency is updated (e.g. boost) then feedstocks are rebuilt
automatically, but ONLY if our packages are in conda-forge, which they
are not. To emulate this, Valerio bumps all the feedstock build
numbers for each release (even ndarray because it depends on boost) so
they will get built against the latest versions in conda-forge.

separate feedstock for py27 since everything is pinned in the recipe:
https://github.com/slac-lcls/psana1-py2-feedstock
recipe/conda_build_config.yaml overrides the equivalent in the conda forge
pinning file (most recent version pulled automatically from conda-forge)

in recipe/conda_build_config.yaml (ONLY for psana1-py2-feedstock),
zip_keys: override specification for the 3 special packages python,
python_impl, numpy. conda-forge people helped specify these, in
particular Billy Poon.

feedstocks:
libpressio-feedstock psana1-py2-feedstock sz-feedstock
ndarray-psana-feedstock psgeom-feedstock xtcav2-feedstock
ndarray-psana-py2-feedstock psocake-feedstock
psana1-feedstock stdcompat-feedstock

libpressio,sz,stdcompat are all needed by chuck/sz.

update build numbers for all the above for every psana1 release.

Checklist for building psana1 environments from feedstocks

As psreldev on psbuld-rhel7:

  • source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
  • conda activate conda_build
  • cd /cds/sw/ds/ana/conda1/manage-feedstocks/
  • bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.11 --basedir `pwd`

As normal user:

  • cd /cds/sw/ds/ana/conda1/manage-feedstocks/downloads/anarel
  • cp psana-conda-4.0.11.tar.gz /reg/g/psdm/web/swdoc/tutorials
  • sha256sum psana-conda-4.0.11.tar.gz <copy the checksum>

Activate an environment that contains the 'conda smithy' package. For example:

conda activate /cds/home/v/valmar/.conda/envs/conda_forge_build3.9

Then for each of these feedstock (more or less in this order):

            github.com/slac-lcls/sz-feedstock
            github.com/slac-lcls/stdcompat-feedstock
            github.com/slac-lcls/ndarray-psana-feedstock
            github.com/slac-lcls/ndarray-psana-py2-feedstock
            github.com/slac-lcls/libpressio-feedstock
            github.com/slac-lcls/xtcav2-feedstock
            github.com/slac-lcls/psgeom-feedstock
            github.com/slac-lcls/psocake-feedstock
            github.com/slac-lcls/psana1-py2-feedstock           
            github.com/slac-lcls/psana1-feedstock            

As normal user:

  • Modify recipe/meta.yaml
    • Check if new version of packages has been released 
    • If yes, bump up version, update sha256 checksum and reset build number
    • Otherwise, bump up build number
  • Commit
  • conda smithy rerender -c auto
  • Push

After all packages have finished building on TravisCI,

As psreldev:

  • source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
  • cd /cds/sw/ds/ana/conda1/manage-feedstocks/jenkins/
  • conda env create -n ana-4.0.11 --file ana-env-py2.yaml
  • conda env create -n ana-4.0.11-py3 --file ana-env-py3.yaml

LCLS2

checkout the 20 feedstock packages (gitHub.com/slac-lcls/): 

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --clone

After these repos are cloned, you can use this to do commands in all repos:

python /cds/sw/ds/ana/conda2/manage/bin/feedstock.py --cmd "pwd; git pull --rebase"

Either update version, or update the build number in each meta.yaml.  conda-forge enforces manual maintenance of version numbers and sha256 hashes (no automatic determination from latest GitHub tag).  If version has changed, compute new sha256 from .tar.gz from GitHub with:

sha256sum 1.1.7.tar.gz

For a new version, remember to reset build number to zero.

Commit the code.

Need an environment where "conda-smithy" is installed (we have put this in psrel's conda_build env).  After making changes to versions/sha256/buildnumber, then "git commit -a -m".  Then rerender with "conda smithy rerender -c auto", then push.  Watch the status of the builds at https://travis-ci.com

Order of the travis builds matters because of conda's "build:, host:" section dependencies (run-time dependencies do not affect this order).  We believe pure-python packages should go in the first wave. These are the waves of builds that can be launched in parallel:

  • libnl, libnl3, roentdek, amityping, prometheus-cpp, libusdusb4, psmon, networkfox, rogue, epix, lcls2_timetool, cameralink-gateway, lcls2-pgp-pcie-apps, xtcdata, ami
  • rmda-core (depends on libnl), psalg (depends on xtcdata)
  • fabric (depends rmda-core), psana (depends on psalg)
  • psdaq (depend on psalg, fabric)

If a package is built for python build matrix tells it to build for the officially supported conda-forge versions (3.6,3.7,3.8).

If we were in conda-forge officially, they take care of these build-order dependencies (but not the versions/sha256/buildnumber).  They may have a bot that tells you that the version was updated, but we still have to update the version number by hand. We haven't gone the conda-forge route because (1) we don't know how much work it is, and (2) cannot upload source .tar.gz to conda-forge, usually get it from GitHub, but the rogue packages are not public.

To create the environment as psrel:

conda create --name ps-N.N.N env_create.yaml (devel env)

conda create --name ps-N.N.N prod_create.yaml (prod env)

  • No labels