Notes about building psana releases.
psana1
Assemble all packages source code in tar.gz file
pslogin
ssh psbuild-rhel7-01 -l psreldev
See details in ~/.bash_history or ~/.k5login
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
conda activate conda_build
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
ls scratch/ # to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` --tagsfile psana-conda-svn-pkgs[-test]
command assembles all the source code from the tags into a .tar.gz file
--basedir`pwd` is converted to /cds/sw/ds/ana/conda1/manage/
< config/psana-conda-svn-pkgs # contains the list of packages with source code
> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56
Accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
added bin/ana-rel-admin: sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
why do not copy psana-conda-4.0.56.tar.gz to /reg/g/psdm/web/swdoc/tutorials/ in bin/ana-rel-admin?
Conda build
ssh pslogin (not as psreldev)
then copy it here, because psreldev does not have permission !!!!???
cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
>>> because later it is used in psana1-feedstock/recipe/meta.yaml as
url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz
ssh psbuild-rhel7-01 -l psreldevcd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cd psana1-feedstock
Set environment:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
OR:
sha256sum /reg/g/psdm/web/swdoc/tutorials/psana-conda-4.0.56.tar.gz
443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
{% set version = '4.0.56' %}
...
sha256: 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
...
extra:
recipe-maintainers:
- valmar
- chrisvam
- dubrovin
update it in github https://github.com/slac-lcls/psana1-feedstock/blob/main/recipe/meta.yaml
do it on github directly, because psreldev does not have permission !!!!???
(conda_build) [psreldev@psbuild-rhel7-01 manage]$ git config user.name
Valerio Mariani
Fixing issue in conda build with permission denied
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ conda create -n local_conda_build python conda-build conda-validate anaconda-client
Collecting package metadata (current_repodata.json): failed
NotWritableError: The current user does not have write permissions to a required path.
path: /cds/sw/ds/ana/conda2-v2/inst/pkgs/cache/18414ddb.json
uid: 14427
gid: 1109
If you feel that permissions on this path are set incorrectly, you can manually
change them by executing
$ sudo chown 14427:1109 /cds/sw/ds/ana/conda2-v2/inst/pkgs/cache/18414ddb.json
In general, it's not advisable to use 'sudo conda'.
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$
Command conda build
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
xorg-xf86vidmodeproto: 2.3.1-h7f98852_1002 conda-forge
xorg-xproto: 7.0.31-h7f98852_1007 conda-forge
xz: 5.2.6-h166bdaf_0 conda-forge
zeromq: 4.3.5-h59595ed_0 conda-forge
zipp: 3.17.0-pyhd8ed1ab_0 conda-forge
zlib: 1.2.13-hd590300_5 conda-forge
zstd: 1.5.5-hfc55251_0 conda-forge
Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/cds/home/p/psreldev/conda-bld/psana_1700187012881/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
export SRC_DIR=/cds/home/p/psreldev/conda-bld/psana_1700187012881/test_tmp
+ psana -h
Usage: psana [options] [dataset ...]
Available options:
{-h|-?|--help } print help message
{-v|--verbose } (incr) verbose output, multiple allowed (initial: 0)
{-q|--quiet } (incr) quieter output, multiple allowed (initial: 2)
{-b|--calib-dir } path calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "")
{-c|--config } path configuration file, by default use psana.cfg if it exists (default: "")
{-e|--experiment } string experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "")
{-j|--job-name } string job name, default is to generate from input file names (default: "")
{-m|--module } name module name, more than one possible
{-n|--num-events } number maximum number of events to process, 0 means all (default: 0)
{-s|--skip-events} number number of events to skip (default: 0)
{-p|--num-cpu } number number greater than 0 enables multi-processing (default: 0)
{-o|--option } string configuration options, format: module.option[=value]
Positional parameters:
dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...)
+ exit 0
Resource usage statistics from testing psana:
Process count: 1
CPU time: Sys=0:00:00.1, User=-
Memory: 1.4M
Disk usage: 12B
Time elapsed: 0:00:02.4
TEST END: /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO:conda_build.utils:Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO conda_build.utils:shutil_move_more_retrying(2075): Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
INFO:conda_build.utils:shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
INFO conda_build.utils:shutil_move_more_retrying(2082): shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:
anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
anaconda_upload is not set. Not uploading wheels: []
####################################################################################
Resource usage summary:
Total time: 1:22:33.6
CPU usage: sys=0:03:04.5, user=0:27:14.5
Maximum memory usage observed: 1.6G
Total disk usage observed (not including envs): 285.6M
####################################################################################
Source and build intermediates have been left in /cds/home/p/psreldev/conda-bld.
There are currently 1 accumulated.
To remove them, you can run the ```conda build purge``` command
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$
Upload the file to anaconda lcls-i channel
anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "chrisvam" as upload username
Processing '/cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2'
Detecting file type...
File type is "conda"
Extracting conda package attributes for upload
Creating package "psana"
Creating release "4.0.56"
Uploading file "chrisvam/psana/4.0.56/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2"
uploaded 18698 of 18698Kb: 100.00% ETA: 0.0 minutes
Upload complete
conda package located at:
https://anaconda.org/chrisvam/psana
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$
Questions
- see Qs above interlaced in red
- removing 7 packages for modular psana, like in config/psana-conda-svn-pkgs-test (saves ~1.5% of space in tar.gz.)
- location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=.. : manage/scratch/data_test with fixed path /cds/sw/ds/ana/conda1/manage/scratch/data_test
- in this case data_test is ignored in
tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56
- subdir=/tmp : /tmp/data_test
- conda build takes 1h22m. How to speed-up conda build and jamp directly to scons test-Detector?
References