Notes about building psana releases.

psana1

Assemble all packages source code in tar.gz file

pslogin
ssh psbuild-rhel7-01 -l psreldev

Set environment:

source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh

conda activate conda_build

cd /cds/sw/ds/ana/conda1/manage

look at subdirectory scratch/ to see what is the latest release, e.g. psana-conda-4.0.55, then create next:

bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` --tagsfile psana-conda-svn-pkgs[-test]

Algorithm of ana-rel-admin

Command bin/ana-rel-admin ... assembles all the source code from github and svn archives with the latest tags into a .tar.gz file

--basedir`pwd` is converted to /cds/sw/ds/ana/conda1/manage/

input: < config/psana-conda-svn-pkgs # contains the list of packages with source code

output:

> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn

> git clone git@github.com:lcls-psana/data_test.git /tmp/data_test
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56

> dumps at the end sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d

Command bin/ana-rel-admin creates file accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz

sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz

> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d

Conda build

ssh pslogin (not as psreldev yet...)

ssh psbuild-rhel7-01 -l psreldev

cd git/psana1-feedstock/

Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml fields for set version and sha256:

edit psana1-feedstock/recipe/meta.yaml

{% set version = '4.0.56' %}
 ...
 sha256: 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
 ...

Set environment:

source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh # OR conda deactivate after previous conda_build

conda activate conda_build_py3

Execute command:

conda build -c lcls-i -c conda-forge recipe

> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2

conda build - the tail of output

    xorg-xf86vidmodeproto:     2.3.1-h7f98852_1002          conda-forge
    xorg-xproto:               7.0.31-h7f98852_1007         conda-forge
    xz:                        5.2.6-h166bdaf_0             conda-forge
    zeromq:                    4.3.5-h59595ed_0             conda-forge
    zipp:                      3.17.0-pyhd8ed1ab_0          conda-forge
    zlib:                      1.2.13-hd590300_5            conda-forge
    zstd:                      1.5.5-hfc55251_0             conda-forge

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
export PREFIX=/cds/home/p/psreldev/conda-bld/psana_1700187012881/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl
export SRC_DIR=/cds/home/p/psreldev/conda-bld/psana_1700187012881/test_tmp
+ psana -h

Usage: psana [options] [dataset ...]

  Available options:
    {-h|-?|--help    }         print help message
    {-v|--verbose    } (incr)  verbose output, multiple allowed (initial: 0)
    {-q|--quiet      } (incr)  quieter output, multiple allowed (initial: 2)
    {-b|--calib-dir  } path    calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "")
    {-c|--config     } path    configuration file, by default use psana.cfg if it exists (default: "")
    {-e|--experiment } string  experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "")
    {-j|--job-name   } string  job name, default is to generate from input file names (default: "")
    {-m|--module     } name    module name, more than one possible
    {-n|--num-events } number  maximum number of events to process, 0 means all (default: 0)
    {-s|--skip-events} number  number of events to skip (default: 0)
    {-p|--num-cpu    } number  number greater than 0 enables multi-processing (default: 0)
    {-o|--option     } string  configuration options, format: module.option[=value]

  Positional parameters:
    dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...)

+ exit 0

Resource usage statistics from testing psana:
   Process count: 1
   CPU time: Sys=0:00:00.1, User=-
   Memory: 1.4M
   Disk usage: 12B
   Time elapsed: 0:00:02.4

TEST END: /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO:conda_build.utils:Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
INFO conda_build.utils:shutil_move_more_retrying(2075): Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop'
shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
INFO:conda_build.utils:shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
INFO conda_build.utils:shutil_move_more_retrying(2082): shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop)
# Automatic uploading is disabled
# If you want to upload package(s) to anaconda.org later, type:

anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2

# To have conda build upload to anaconda.org automatically, use
# $ conda config --set anaconda_upload yes
anaconda_upload is not set.  Not uploading wheels: []
####################################################################################
Resource usage summary:

Total time: 1:22:33.6
CPU usage: sys=0:03:04.5, user=0:27:14.5
Maximum memory usage observed: 1.6G
Total disk usage observed (not including envs): 285.6M

####################################################################################
Source and build intermediates have been left in /cds/home/p/psreldev/conda-bld.
There are currently 1 accumulated.
To remove them, you can run the ```conda build purge``` command
(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$

Upload the file to anaconda lcls-i channel

anaconda upload -u lcls-i /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2

anaconda upload messages

(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "chrisvam" as upload username
Processing '/cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2'
Detecting file type...
File type is "conda"
Extracting conda package attributes for upload
Creating package "psana"
Creating release "4.0.56"
Uploading file "chrisvam/psana/4.0.56/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2"
 uploaded 18698 of 18698Kb: 100.00% ETA: 0.0 minutes
Upload complete

conda package located at:
https://anaconda.org/chrisvam/psana

(conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$

Create new environment

goal: create a real 4.0.57

conda create --name psana-conda-4.0.57 --clone psana-conda-4.0.56
conda deactivate
conda activate psana-conda-4.0.57
conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.57

have to do this on both old-psana and new-s3df as psreldev

Extra operetions

See details in ~/.bash_history or ~/.k5login

cd /cds/sw/ds/ana/conda1/

if ./manage is not available

git clone git@github.com:slaclab/anarel-manage.git manage

cd manage

update it if necessary

git pull --rebase

conda create -n conda_build_py3 python=3.9 anaconda

conda env remove --name conda_build_py3

see /cds/sw/ds/ana/conda1/inst/envs/ OR conda info --envs

ssh psbuild-rhel7-01 -l psreldev

cd git/psana1-feedstock/

sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz

> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d

~psreldev/git/psana1-feedstock/recipe/meta.yaml

cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock

Set environment (chris):

source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build

OR:

source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh

conda create -n conda_build_py3 python conda-build anaconda-client

conda activate conda_build_py3

ssh psbuild-rhel7-01 -l psreldev

cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock

cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/

copy here because psreldev does not have permission !!!!???
because later it is used in psana1-feedstock/recipe/meta.yaml as url: https://pswww.slac.stanford.edu/swdoc/tutorials/{{ name }}-conda-{{ version }}.tar.gz

Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'

Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available

Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client

scons test-Detector

git repository data_test_access cloned with all other psana packages
location for data_test can be controlled in config/psana-conda-svn-pkgs:
- subdir=/tmp - clones this package as /tmp/data_test
data_test_access is included but data_test is ignored in tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56

Build on s3df

Initialization for build on s3df

IT IS ALREADY DONE ONCE

create conda_build_svn environment

s3dfnx - nx - server on s3df connects to dubrovin@sdflogin002

. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

conda create -n conda_build_svn svn anaconda-client

clone anarel-manage

cd .../con-build OR cd /sdf/group/lcls/ds/ana/sw/conda1/
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

git clone git@github.com:slaclab/anarel-manage.git

make github seen from psana nodes

ema ~/.gitconfig

\[http\] - remove

Run bin/ana-rel-admin on s3df

login as myself

> s3dfnx ends up as dubrovin@sdflogin002

> psana ... because /afs/slac/g/pcds/svn/pdsdata is seen from psana nodes only... (Wilko)

cd <path-to>/anarel-manage

login as psreldev

> s3dfnx ends up as dubrovin@sdflogin002

ssh psana -l psreldev

cd /sdf/group/lcls/ds/ana/sw/conda1/manage

conda activate and run ana-rel-admin

source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

conda deactivate

conda activate conda_build_svn

bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd`

Conda build

Everything works on s3df psana node under login psreldev because it needs some disk space... Other than that it can be built under any. other user.

Clone psana1-feedstock

IT IS ALREADY DONE ONCE for psreldev

> s3dflogin

ssh psana -l psreldev

cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock

Installing conda-build in the conda base

source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

conda deactivate # > base

conda activate base conda install conda-build
conda install conda-verify # also works in the base

Set environment

source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

conda deactivate

conda activate conda_build_svn

Run conda build

cd ~/git/psana1-feedstock/

Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml fields for set version and sha256

conda build -c lcls-i -c conda-forge recipe

References

Building Conda Packages And Releases
conda-cheatsheet.pdf
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
/reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
Running at S3DF

Page tree

Building psana release

psana1

Assemble all packages source code in tar.gz file

Algorithm of ana-rel-admin

Conda build

Upload the file to anaconda lcls-i channel

Create new environment

Extra operetions

scons test-Detector

Build on s3df

Initialization for build on s3df

create conda_build_svn environment

clone anarel-manage

make github seen from psana nodes

Run bin/ana-rel-admin on s3df

login as myself

login as psreldev

conda activate and run ana-rel-admin

Conda build

Clone psana1-feedstock

Installing conda-build in the conda base

Set environment

Run conda build

References