Page History
...
pslogin
ssh psbuild-rhel7-01 -l psreldev
See details in ~/.bash_history or ~/.k5login
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
conda
...
activate conda_build
...
cd
conda env remove --name conda_build_py3
...
/cds/sw/ds/ana/conda1/
...
manage
ln scratch/ # to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin
...
cpnda activate conda_build_py3
conda activate conda_build
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
ls scratch/ # to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` `pwd` --tagsfile psana-conda-svn-pkgs[-test]
...
> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56
> dumps at the end sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Command bin/ana-rel-admin creates file accessible through Accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gzadded dump in bin/ana-rel-admin: sha256sum ...
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
...
Conda build
ssh pslogin (not as psreldev yet...)
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256:
Code Block | ||||
---|---|---|---|---|
| ||||
{% set version = '4.0.56' %}
...
sha256: 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
... |
Set environment:
source
why psana-conda-4.0.56.tar.gz is needed under /reg/g/psdm/web/swdoc/tutorials/ ?
why do not copy psana-conda-4.0.56.tar.gz to /reg/g/psdm/web/swdoc/tutorials/ in bin/ana-rel-admin?
Conda build
ssh pslogin (not as psreldev yet...)
cp /cds/sw/ds/ana/conda1/manageinst/downloadsetc/anarelprofile.d/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
- copy here because psreldev does not have permission !!!!???
- because later it is used in psana1-feedstock/recipe/meta.yaml as url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz
ssh psbuild-rhel7-01 -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cd psana1-feedstock
Set environment:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz OR:
sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.56.tar.gz
443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Code Block | ||||
---|---|---|---|---|
| ||||
{% set version = '4.0.56' %}
...
sha256: 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
...
extra:
recipe-maintainers:
- valmar
- chrisvam
- dubrovin |
update it in github https://github.com/slac-lcls/psana1-feedstock/blob/main/recipe/meta.yaml
do it on github directly, because psreldev does not have permission !!!!???
(conda_build) [psreldev@psbuild-rhel7-01 manage]$ git config user.name
Valerio Mariani
Fixing issue in conda build with permission denied
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
Code Block | ||||
---|---|---|---|---|
| ||||
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ conda create -n local_conda_build python conda-build conda-validate anaconda-client
Collecting package metadata (current_repodata.json): failed
NotWritableError: The current user does not have write permissions to a required path.
path: /cds/sw/ds/ana/conda2-v2/inst/pkgs/cache/18414ddb.json
uid: 14427
gid: 1109
If you feel that permissions on this path are set incorrectly, you can manually
change them by executing
$ sudo chown 14427:1109 /cds/sw/ds/ana/conda2-v2/inst/pkgs/cache/18414ddb.json
In general, it's not advisable to use 'sudo conda'.
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
Command conda build
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
.sh # OR conda deactivate after previous conda_build
conda activate conda_build_py3
Execute command:
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Code Block | ||||
---|---|---|---|---|
| ||||
xorg-xf86vidmodeproto: 2.3.1-h7f98852_1002 conda-forge
xorg-xproto: 7.0.31-h7f98852_1007 conda-forge
xz: 5.2.6-h166bdaf_0 conda-forge
zeromq: 4.3.5-h59595ed_0 conda-forge
zipp: 3.17.0-pyhd8ed1ab_0 conda-forge
zlib: 1.2.13-hd590300_5 conda-forge
zstd: 1.5.5-hfc55251_0 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/cds/home/p/psreldev/conda-bld/psana_1700187012881/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
export SRC_DIR=/cds/home/p/psreldev/conda-bld/psana_1700187012881/test_tmp
+ psana -h
Usage: psana [options] [dataset ...]
Available options:
{-h|-?|--help } print help message
{-v|--verbose } (incr) verbose output, multiple allowed (initial: 0)
{-q|--quiet } (incr) quieter output, multiple allowed (initial: 2)
{-b|--calib-dir } path calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "")
{-c|--config } path configuration file, by default use psana.cfg if it exists (default: "")
{-e|--experiment } string experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "")
{-j|--job-name } string job name, default is to generate from input file names (default: "")
{-m|--module } name module name, more than one possible
{-n|--num-events } number maximum number of events to process, 0 means all (default: 0)
{-s|--skip-events} number number of events to skip (default: 0)
{-p|--num-cpu } number number greater than 0 enables multi-processing (default: 0)
{-o|--option } string configuration options, format: module.option[=value]
Positional parameters:
dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...)
+ exit 0
Resource usage statistics from testing psana:
Process count: 1
CPU time: Sys=0:00:00.1, User=-
Memory: 1.4M
Disk usage: 12B
Time elapsed: 0:00:02.4
TEST END: /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO:conda_build.utils:Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO conda_build.utils:shutil_move_more_retrying(2075): Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
INFO:conda_build.utils:shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest= | ||||
Code Block | ||||
| ||||
xorg-xf86vidmodeproto: 2.3.1-h7f98852_1002 conda-forge xorg-xproto: 7.0.31-h7f98852_1007 conda-forge xz: 5.2.6-h166bdaf_0 conda-forge zeromq: 4.3.5-h59595ed_0 conda-forge zipp: 3.17.0-pyhd8ed1ab_0 conda-forge zlib: 1.2.13-hd590300_5 conda-forge zstd: 1.5.5-hfc55251_0 conda-forge Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done export PREFIX=/cds/home/p/psreldev/conda-bld/psana_1700187012881/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl export SRC_DIR=/cds/home/p/psreldev/conda-bld/psana_1700187012881/test_tmp + psana -h Usage: psana [options] [dataset ...] Available options: {-h|-?|--help } print help message {-v|--verbose } (incr) verbose output, multiple allowed (initial: 0) {-q|--quiet } (incr) quieter output, multiple allowed (initial: 2) {-b|--calib-dir } path calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "") {-c|--config } path configuration file, by default use psana.cfg if it exists (default: "") {-e|--experiment } string experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "") {-j|--job-name } string job name, default is to generate from input file names (default: "") {-m|--module } name module name, more than one possible {-n|--num-events } number maximum number of events to process, 0 means all (default: 0) {-s|--skip-events} number number of events to skip (default: 0) {-p|--num-cpu } number number greater than 0 enables multi-processing (default: 0) {-o|--option } string configuration options, format: module.option[=value] Positional parameters: dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...) + exit 0 Resource usage statistics from testing psana: Process count: 1 CPU time: Sys=0:00:00.1, User=- Memory: 1.4M Disk usage: 12B Time elapsed: 0:00:02.4 TEST END: /cds/home/p/psreldev/conda-bld/linux-64/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1.tar.bz2 Renaming work directory '_linux-64_main_build_loop) INFO conda_build.utils:shutil_move_more_retrying(2082): shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to ', dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' INFO:conda_build.utils:Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to ') # Automatic uploading is disabled # If you want to upload package(s) to anaconda.org later, type: anaconda upload /cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_linux-64/psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' INFO conda_build.utils:shutil_move_more_retrying(2075): Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=.tar.bz2 # To have conda build upload to anaconda.org automatically, use # $ conda config --set anaconda_upload yes anaconda_upload is not set. Not uploading wheels: [] #################################################################################### Resource usage summary: Total time: 1:22:33.6 CPU usage: sys=0:03:04.5, user=0:27:14.5 Maximum memory usage observed: 1.6G Total disk usage observed (not including envs): 285.6M #################################################################################### Source and build intermediates have been left in /cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) INFO:conda_build.utils:shutil.move(work)=. There are currently 1 accumulated. To remove them, you can run the ```conda build purge``` command (conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
Upload the file to anaconda lcls-i channel
anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Code Block | ||||
---|---|---|---|---|
| ||||
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ anaconda upload _1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_linux-64/psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) INFO conda_build.utils:shutil_move_more_retrying(2082): shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=.tar.bz2 Using Anaconda API: https://api.anaconda.org Using "chrisvam" as upload username Processing '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_linux-64/psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) # Automatic uploading is disabled # If you want to upload package(s) to anaconda.org later, type: anaconda upload /cds/home/p/psreldev/conda-bld.tar.bz2' Detecting file type... File type is "conda" Extracting conda package attributes for upload Creating package "psana" Creating release "4.0.56" Uploading file "chrisvam/psana/4.0.56/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2 " # Touploaded have conda build upload to anaconda.org automatically, use # $ conda config --set anaconda_upload yes anaconda_upload is not set. Not uploading wheels: [] #################################################################################### Resource usage summary: Total time: 1:22:33.6 CPU usage: sys=0:03:04.5, user=0:27:14.5 Maximum memory usage observed: 1.6G Total disk usage observed (not including envs): 285.6M #################################################################################### Source and build intermediates have been left in /cds/home/p/psreldev/conda-bld. There are currently 1 accumulated. To remove them, you can run the ```conda build purge``` command (conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
Upload the file to anaconda lcls-i channel
anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Code Block | ||||
---|---|---|---|---|
| ||||
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "chrisvam" as upload username
Processing '/cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2'
Detecting file type...
File type is "conda"
Extracting conda package attributes for upload
Creating package "psana"
Creating release "4.0.56"
Uploading file "chrisvam/psana/4.0.56/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2"
uploaded 18698 of 18698Kb: 100.00% ETA: 0.0 minutes
Upload complete
conda package located at:
https://anaconda.org/chrisvam/psana
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
Create new environment
goal: create a real 4.0.57
...
18698 of 18698Kb: 100.00% ETA: 0.0 minutes
Upload complete
conda package located at:
https://anaconda.org/chrisvam/psana
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
Create new environment
goal: create a real 4.0.57
conda create --name psana-conda-4.0.57 --clone psana-conda-4.0.56
conda deactivate
conda activate psana-conda-4.0.57
conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.57
have to do this on both old-psana and new-s3df as psreldev
Extra operetions
See details in ~/.bash_history or ~/.k5login
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
conda create -n conda_build_py3 python=3.9 anaconda
conda env remove --name conda_build_py3
see /cds/sw/ds/ana/conda1/inst/envs/ OR conda info --envs
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
~psreldev/git/psana1-feedstock/recipe/meta.yaml
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
Set environment (chris):
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
OR:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
conda create -n conda_build_py3 python conda-build anaconda-client
conda activate conda_build_py3
ssh psbuild-rhel7-01 -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
- copy here because psreldev does not have permission !!!!???
- because later it is used in psana1-feedstock/recipe/meta.yaml as url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
scons test-Detector
- git repository data_test_access cloned with all other psana packages
- location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=/tmp - clones this package as /tmp/data_test
- data_test_access is included but
Questions
- see Qs above interlaced in red
- removing 7 packages for modular psana, like in config/psana-conda-svn-pkgs-test (saves ~1.5% of space in tar.gz.)
- location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=.. : manage/scratch/data_test with fixed path /cds/sw/ds/ana/conda1/manage/scratch/data_test
- subdir=/tmp : /tmp/data_test in these cases data_test is ignored in
tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56
conda build takes 1h22m. How to speed-up conda build and jamp directly to scons test-Detector?References
- Building Conda Packages And Releases
- conda-cheatsheet.pdf
- https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
- /reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh