LCLS1

Information from Valerio on Jan. 27, 2021.

NOTE: the feedstock approach is currently deprecated because conda forge moved too fast for us and there were often problems in the release.  See "Additions for Pinned Approach" section below for modifications for LCLS1 "pinned approach".

New Pinned-Approach

Current as of Oct, 9, 2023

Valerio writes: It's not different from the psana2 approach at all.  First you need to package the source in a tar.gz like we always did, as shown in the "Checklist for building psana1 environments from feedstocks" section below:(see also checklist below for which env to activate and how to access ana-rel-admin)

cd /cds/sw/ds/ana/conda1/manage/
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.52 --basedir `pwd`

...

generate new checksum with: sha256sum .tar.gz like this:

sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.10.tar.gz

git push launches the build on GitHub (using GitHub Actions) when it finishes it uploads it
to anaconda.org/lcls-i

this is our real control of the build process (e.g. the GitHub Actions Workflow yaml file gets
generated automatically)

The py2 and py3 psana recipes have now the pinning file (conda_build_config.yaml) in them. You just need to update the meta.yaml file, run "conda build recipe" (did this as user "cpo", but could also do as psreldev, although psreldev might need "anaconda login" as another user so "anaconda upload" will work) upload and build the environment from the "pinned" YAML file. 

The py3 recipe is in https://github.com/slac-lcls/psana1-feedstock/blob/main/conda-forge.yml

if the above file is modified have to run "conda smithy rerender -c
auto" "-c auto" means automatically make a git commit. but still have
to push to git.

There are a few other cases in which the feedstock needs to be rerendered.
See the conda-forge documentation:

https://conda-forge.org/docs/maintainer/updating_pkgs.html#when-to-rerender

If no relevant changes have taken place, the command will simply not create a
commit. so maybe it is a good practice to run it every time a feedstock is
updated.

"targets" first entry: the conda channel to upload to, second entry is
called a "label" (e.g. a devel and a stable). conda forge always looks
for "main" label.

then "conda create" as usual (as seen in the jenkins CI).

for testing the feedstock, can run the run_docker_build.sh script locally
but requires docker to be installed. The file  "build_locally.py" included in
the feedstock takes care of running run_docker_build.sh

Valerio uses this to test the feedstock on his laptop.

If any dependency is updated (e.g. boost) then feedstocks are rebuilt
automatically, but ONLY if our packages are in conda-forge, which they
are not. To emulate this, Valerio bumps all the feedstock build
numbers for each release (even ndarray because it depends on boost) so
they will get built against the latest versions in conda-forge.

separate feedstock for py27 since everything is pinned in the recipe:
https://github.com/slac-lcls/psana1-py2-feedstock
recipe/conda_build_config.yaml overrides the equivalent in the conda forge
pinning file (most recent version pulled automatically from conda-forge)

in recipe/conda_build_config.yaml (ONLY for psana1-py2-feedstock),
zip_keys: override specification for the 3 special packages python,
python_impl, numpy. conda-forge people helped specify these, in
particular Billy Poon.

feedstocks:
libpressio-feedstock psana1-py2-feedstock sz-feedstock
ndarray-psana-feedstock psgeom-feedstock ztcav-py2-feedstock xtcav2-feedstock
ndarray-psana-py2-feedstock psocake-py2-feedstock psocake-feedstock
psana1-feedstock stdcompat-feedstock

libpressio,sz,stdcompat are all needed by chuck/sz.

update build numbers for all the above for every psana1 release.

Checklist for building psana1 environments from feedstocks

As psreldev on psbuld-rhel7:

...

while the py2 recipe is in https://github.com/slac-lcls/psana1-py2-feedstock.

cpo thinks the psana1 recipes (both py2 and py3) need to be built with the python3 conda_build env from LCLS2 (source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh).  If you don't do this I think we get this error from "conda build recipe/" since conda_build env in LCLS1 is py2.  built the recipes as user cpo.

TypeError: apply_pin_expressions() argument after ** must be a mapping, not unicode

Upload the file to conda-forge with:

anaconda upload -u lcls-i <.tar.bz2>

The yaml files for creating the environment are in /cds/sw/ds/ana/conda1/manage/jenkins/.

To create the env's without changing the pinnings, clone the previous environment and install the newly upload psana version like this:

conda install -c lcls-i psana=4.0.44 --experimental-solver=libmamba

To create new pinnings (which may require also changing conda_build_config.yaml in the py2/py3 recipe repos) you can create two env's like this:

source /reg/g/psdm/etc/psconda.sh -v2 (v2 to get experimental mamba solver)
conda env create --name ana-4.0.44-py3 --experimental-solver=libmamba --file=/cds/sw/ds/ana/conda1/manage/jenkins/ana-env-py3.yaml 
conda env create --name ana-4.0.44 --experimental-solver=libmamba --file=/cds/sw/ds/ana/conda1/manage/jenkins/ana-env-py2.yaml

Deprecated Instructions

Information from Valerio on Jan. 27, 2021.

NOTE: the feedstock approach is currently deprecated because conda forge moved too fast for us and there were often problems in the release.  See "Additions for Pinned Approach" section below for modifications for LCLS1 "pinned approach".

(see also checklist below for which env to activate and how to access ana-rel-admin)

...

cd /cds/sw/ds/ana/conda1/manage/
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.

...

52 --basedir `pwd`

As normal user:

...

cp .tar.gz (from manage/downloads/anarel

...

/) to where GitHub can see it:
/reg/g/psdm/web/swdoc/tutorials

...

/

psreldev can't write to the above, do as user "cpo" or "valmar"

checkout github feedstock repository:
many feedstocks (all in slac-lcls) e.g.
https://github.com/slac-lcls/psana1-feedstock

update version and checksum in

https://

Activate an environment that contains the 'conda smithy' package. For example:

conda activate /cds/home/v/valmar/.conda/envs/conda_forge_build3.9

Then for each of these feedstock (more or less in this order):

            github.com/slac-lcls/szpsana1-feedstock
            github.com/slac-lcls/stdcompat-feedstock
            github.com/slac-lcls/ndarray-psana-feedstock
            github.com/slac-lcls/ndarray-psana-py2-feedstock
            github.com/slac-lcls/libpressio-feedstock
/blob/main/recipe/meta.yaml

generate new checksum with: sha256sum .tar.gz

sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.10.tar.gz

git push launches the build on GitHub (using GitHub Actions) when it finishes it uploads it
to anaconda.org/lcls-i

this is our real control of the build process (e.g. the GitHub Actions Workflow yaml file gets
generated automatically)

https://            github.com/slac-lcls/xtcav2psana1-feedstock
            github.com/slac-lcls/psgeom-feedstock
            github.com/slac-lcls/psocake-feedstock
            github.com/slac-lcls/psana1-py2-feedstock           
            github.com/slac-lcls/psana1-feedstock            

As normal user:

• Modify recipe/meta.yaml
  • Check if new version of packages has been released 
  • If yes, bump up version, update sha256 checksum and reset build number
  • Otherwise, bump up build number
• Commit
• conda smithy rerender -c auto
• Push

After all packages have finished building on GitHub (Unfortunately, GitHub Actions do not provide a "dashboard" to check if the packages have been built or when they are finished. There are only two ways to check this: 1) Manually check every feedstock repository (Under the "Actions" tab) 2) Wait for either the package to appear on anaconda.org/lcls-i or for an email notifying a failure to arrive):

As psreldev:

• source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
• cd /cds/sw/ds/ana/conda1/manage/jenkins/
• conda env create -n ana-4.0.11 --file ana-env-py2.yaml
• conda env create -n ana-4.0.11-py3 --file ana-env-py3.yaml

New Pinned-Approach

Valerio writes: It's not different from the psana2 approach at all.  First you need to package the source in a tar.gz like we always did, as shown in the "Checklist for building psana1 environments from feedstocks" section above.  The py2 and py3 psana recipes have now the pinning file (conda_build_config.yaml) in them. You just need to update the meta.yaml file, run "conda build recipe" (did this as user "cpo", but could also do as psreldev, although psreldev might need "anaconda login" as another user so "anaconda upload" will work) upload and build the environment from the "pinned" YAML file. 

The py3 recipe is in https://github.com/slac-lcls/psana1-feedstock while the py2 recipe is in https://github.com/slac-lcls/psana1-py2-feedstock.

...

/blob/main/conda-forge.yml

if the above file is modified have to run "conda smithy rerender -c
auto" "-c auto" means automatically make a git commit. but still have
to push to git.

There are a few other cases in which the feedstock needs to be rerendered.
See the conda-forge documentation:

https://conda-forge.org/docs/maintainer/updating_pkgs.html#when-to-rerender

If no relevant changes have taken place, the command will simply not create a
commit. so maybe it is a good practice to run it every time a feedstock is
updated.

"targets" first entry: the conda channel to upload to, second entry is
called a "label" (e.g. a devel and a stable). conda forge always looks
for "main" label.

then "conda create" as usual (as seen in the jenkins CI).

for testing the feedstock, can run the run_docker_build.sh script locally
but requires docker to be installed. The file  "build_locally.py" included in
the feedstock takes care of running run_docker_build.sh

Valerio uses this to test the feedstock on his laptop.

If any dependency is updated (e.g. boost) then feedstocks are rebuilt
automatically, but ONLY if our packages are in conda-forge, which they
are not. To emulate this, Valerio bumps all the feedstock build
numbers for each release (even ndarray because it depends on boost) so
they will get built against the latest versions in conda-forge.

separate feedstock for py27 since everything is pinned in the recipe:
https://github.com/slac-lcls/psana1-py2-feedstock
recipe/conda_build_config.yaml overrides the equivalent in the conda forge
pinning file (most recent version pulled automatically from conda-forge)

in recipe/conda_build_config.yaml (ONLY for psana1-py2-feedstock),
zip_keys: override specification for the 3 special packages python,
python_impl, numpy. conda-forge people helped specify these, in
particular Billy Poon.

feedstocks:
libpressio-feedstock psana1-py2-feedstock sz-feedstock
ndarray-psana-feedstock psgeom-feedstock ztcav-py2-feedstock xtcav2-feedstock
ndarray-psana-py2-feedstock psocake-py2-feedstock psocake-feedstock
psana1-feedstock stdcompat-feedstock

libpressio,sz,stdcompat are all needed by chuck/sz.

update build numbers for all the above for every psana1 release.

Checklist for building psana1 environments from feedstocks

As psreldev on psbuld-rhel7:

• source /cds/sw/ds/ana/

...

• conda1/

...

• inst/etc/profile.d/conda.sh
• conda activate conda_build
• cd /cds/sw/ds/ana/conda1/manage/
• bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.11 --basedir `pwd`

As normal user:

• cd

TypeError: apply_pin_expressions() argument after ** must be a mapping, not unicode

...

• /cds/sw/ds/ana/conda1/manage/

...

• downloads/

...

To create the env's without changing the pinnings, clone the previous environment and install the newly upload psana version like this:

conda install -c lcls-i psana=4.0.44 --experimental-solver=libmamba

To create new pinnings (which may require also changing conda_build_config.yaml in the py2/py3 recipe repos) you can create two env's like this:

...

• anarel
• cp psana-conda-4.0.11.tar.gz /reg/g/psdm/web/swdoc/tutorials
• sha256sum psana-conda-4.0.11.tar.gz <copy the checksum>

Activate an environment that contains the 'conda smithy' package. For example:

conda activate /cds/home/v/valmar/.conda/envs/conda_forge_build3.9

Then for each of these feedstock (more or less in this order):

            github.com/slac-lcls/sz-feedstock
            github.com/slac-lcls/stdcompat-feedstock
            github.com/slac-lcls/ndarray-psana-feedstock
            github.com/slac-lcls/ndarray-psana-py2-feedstock
            github.com/slac-lcls/libpressio-feedstock
            github.com/slac-lcls/xtcav2-feedstock
            github.com/slac-lcls/psgeom-feedstock
            github.com/slac-lcls/psocake-feedstock
            github.com/slac-lcls/psana1-py2-feedstock           
            github.com/slac-lcls/psana1-feedstock            

As normal user:

• Modify recipe/meta.yaml
  • Check if new version of packages has been released 
  • If yes, bump up version, update sha256 checksum and reset build number
  • Otherwise, bump up build number
• Commit
• conda smithy rerender -c auto
• Push

After all packages have finished building on GitHub (Unfortunately, GitHub Actions do not provide a "dashboard" to check if the packages have been built or when they are finished. There are only two ways to check this: 1) Manually check every feedstock repository (Under the "Actions" tab) 2) Wait for either the package to appear on anaconda.org/lcls-i or for an email notifying a failure to arrive):

As psreldev:

• source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
• cd /cds/sw/ds/ana/conda1/manage/jenkins/
• conda env create -n ana-4.0.11 --file ana-env-

...

• py2.yaml

...

• conda

...

• env

...

• create

...

• -

...

• n ana-4.0.

...

• 11-py3 --file

...

• ana

...

• -env-

...

• py3.yaml

LCLS2 Github Actions Approach (Deprecated)

...