Notes about building psana releases.
psana1
Assemble all packages source code in tar.gz file
pslogin
ssh psbuild-rhel7-01 -l psreldev
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
conda activate conda_build
cd /cds/sw/ds/ana/conda1/manage
look at subdirectory scratch/ to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` --tagsfile psana-conda-svn-pkgs[-test]
Algorithm of ana-rel-admin
Command bin/ana-rel-admin ... assembles all the source code from github and svn archives with the latest tags into a .tar.gz file
--basedir`pwd` is converted to /cds/sw/ds/ana/conda1/manage/
input: < config/psana-conda-svn-pkgs # contains the list of packages with source code
output:
> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn
> git clone git@github.com:lcls-psana/data_test.git /tmp/data_test
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56
> dumps at the end sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Command bin/ana-rel-admin creates file accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Conda build
ssh pslogin (not as psreldev yet...)
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256:
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh # OR conda deactivate after previous conda_build
conda activate conda_build_py3
Execute command:
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Upload the file to anaconda lcls-i channel
anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Create new environment
goal: create a real 4.0.57
conda create --name psana-conda-4.0.57 --clone psana-conda-4.0.56
conda deactivate
conda activate psana-conda-4.0.57
conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.57
have to do this on both old-psana and new-s3df as psreldev
Extra operetions
See details in ~/.bash_history or ~/.k5login
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
conda create -n conda_build_py3 python=3.9 anaconda
conda env remove --name conda_build_py3
see /cds/sw/ds/ana/conda1/inst/envs/ OR conda info --envs
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
~psreldev/git/psana1-feedstock/recipe/meta.yaml
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
Set environment (chris):
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
OR:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
conda create -n conda_build_py3 python conda-build anaconda-client
conda activate conda_build_py3
ssh psbuild-rhel7-01 -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
- copy here because psreldev does not have permission !!!!???
- because later it is used in psana1-feedstock/recipe/meta.yaml as url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
scons test-Detector
- git repository data_test_access cloned with all other psana packages
- location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=/tmp - clones this package as /tmp/data_test
- data_test_access is included but data_test is ignored in
tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56
Build on s3df
Initialization for build on s3df
IT IS ALREADY DONE ONCE
create conda_build_svn environment
s3dfnx - nx - server on s3df connects to dubrovin@sdflogin002
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda create -n conda_build_svn svn anaconda-client
clone anarel-manage
cd .../con-build OR cd /sdf/group/lcls/ds/ana/sw/conda1/
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
git clone git@github.com:slaclab/anarel-manage.git
make github seen from psana nodes
ema ~/.gitconfig
\[http\] - remove
Run bin/ana-rel-admin on s3df
login as myself
> s3dfnx ends up as dubrovin@sdflogin002
> psana ... because /afs/slac/g/pcds/svn/pdsdata is seen from psana nodes only... (Wilko)
cd <path-to>/anarel-manage
login as psreldev
> s3dfnx ends up as dubrovin@sdflogin002
ssh psana -l psreldev
cd /sdf/group/lcls/ds/ana/sw/conda1/manage
conda activate and run ana-rel-admin
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate conda_build_svn
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd`
Command bin/ana-rel-admin creates file
/sdf/group/lcls/ds/ana/sw/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
accessible in ~psreldev/git/psana1-feedstock/recipe/meta.yaml through url:
file:///sdf/group/lcls/ds/ana/sw/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
Conda build
Everything works on s3df psana node under login psreldev because it needs some disk space... Other than that it can be built under any. other user.
Clone psana1-feedstock
IT IS ALREADY DONE ONCE for psreldev
> s3dflogin
ssh psana -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
Installing conda-build in the conda base
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate # > base
conda activate base conda install conda-build
conda install conda-verify # also works in the base
Set environment
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate conda_build_svn
Run conda build
cd ~/git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256
conda build -c lcls-i -c conda-forge recipe
References
- Building Conda Packages And Releases
- conda-cheatsheet.pdf
- https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
- /reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
- Running at S3DF