Notes about building psana releases.
psana1
Assemble all packages source code in tar.gz file
pslogin
ssh psbuild-rhel7-01 -l psreldev
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
conda activate conda_build
cd /cds/sw/ds/ana/conda1/manage
look at subdirectory scratch/ to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` --tagsfile psana-conda-svn-pkgs[-test]
Algorithm of ana-rel-admin
Command bin/ana-rel-admin ... assembles all the source code from github and svn archives with the latest tags into a .tar.gz file
--basedir`pwd` is converted to /cds/sw/ds/ana/conda1/manage/
input: < config/psana-conda-svn-pkgs # contains the list of packages with source code
output:
> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn
> git clone git@github.com:lcls-psana/data_test.git /tmp/data_test
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56
> dumps at the end sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Command bin/ana-rel-admin creates file accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Conda build
ssh pslogin (not as psreldev yet...)
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256:
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh # OR conda deactivate after previous conda_build
conda activate conda_build_py3
Execute command:
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Upload the file to anaconda lcls-i channel
anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Create new environment
goal: create a real 4.0.57
conda create --name psana-conda-4.0.57 --clone psana-conda-4.0.56
conda deactivate
conda activate psana-conda-4.0.57
conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.57
have to do this on both old-psana and new-s3df as psreldev
Extra operetions
See details in ~/.bash_history or ~/.k5login
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
conda create -n conda_build_py3 python=3.9 anaconda
conda env remove --name conda_build_py3
see /cds/sw/ds/ana/conda1/inst/envs/ OR conda info --envs
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
~psreldev/git/psana1-feedstock/recipe/meta.yaml
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
Set environment (chris):
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
OR:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
conda create -n conda_build_py3 python conda-build anaconda-client
conda activate conda_build_py3
ssh psbuild-rhel7-01 -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
- copy here because psreldev does not have permission !!!!???
- because later it is used in psana1-feedstock/recipe/meta.yaml as url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
scons test-Detector
- git repository data_test_access cloned with all other psana packages
- location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=/tmp - clones this package as /tmp/data_test
- data_test_access is included but data_test is ignored in
tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56
Build on s3df
goal1 - run bin/ana-rel-admin on s3df
s3dfnx - nx - server on s3df connects to dubrovin@sdflogin002
cd .../con-build
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda env list
conda activate conda_build
git clone git@github.com:slaclab/anarel-manage.git
cd anarel-manage/
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.58 --basedir `pwd`
>>> missing svn command
conda create -n conda_build_svn svn conda_build anaconda-client
conda deactivate
conda activate conda_build_svn
python -V
Python 3.11.5
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.58 --basedir `pwd`
>>> AssertionError: stderr from cmd=svn ls file:///afs/slac/g/pcds/svn/pdsdata/tags
svn: E170013: Unable to connect to a repository at URL 'file:///afs/slac/g/pcds/svn/pdsdata/tags'
svn: E180001: Unable to open repository 'file:///afs/slac/g/pcds/svn/pdsdata/tags'
On pcds nodes svn repo is file:///afs/slac/g/pcds/svn/pdsdata/tags
Wilko: afs is accessible on the interactive nodes (psana):
sdfiana001> ls /afs/slac/g/pcds/svn/pdsdata
conf/ db/ format hooks/ locks/ README.txt
> sdflogin
> psana
...
conda activate conda_build_svn
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.58 --basedir `pwd`
>>> on sdfiana001 can access svn, but can't access git
ema ~/.gitconfig
[user]
name = msdubrovin
email = msdubrovin@gmail.com
\[http\] - remove
References
- Building Conda Packages And Releases
- conda-cheatsheet.pdf
- https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
- /reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
- Running at S3DF