Notes about building psana releases.
pslogin
ssh psbuild-rhel7-01 -l psreldev
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh
conda activate conda_build
cd /cds/sw/ds/ana/conda1/manage
look at subdirectory scratch/ to see what is the latest release, e.g. psana-conda-4.0.55, then create next:
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd` --tagsfile psana-conda-svn-pkgs[-test]
Command bin/ana-rel-admin ... assembles all the source code from github and svn archives with the latest tags into a .tar.gz file
--basedir`pwd` is converted to /cds/sw/ds/ana/conda1/manage/
input: < config/psana-conda-svn-pkgs # contains the list of packages with source code
output:
> scratch/psana-conda-4.0.56/<packages> # clones of all packages form git or svn
> git clone git@github.com:lcls-psana/data_test.git /tmp/data_test
> scratch/psana-conda-4.0.56/psana-conda-tag # tags extended with version like 'tag': 'V02-00-76'
> scratch/psana-conda-4.0.56/psana-conda-4.0.56/.sit_release # contains psana-conda-4.0.56
> downloads/anarel/psana-conda-4.0.56.tar.gz # archived content of scratch/psana-conda-4.0.56
> dumps at the end sha256sum .../psana-conda-4.0.56.tar.gz > 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
Command bin/ana-rel-admin creates file accessible through url:
file:///cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
ssh pslogin (not as psreldev yet...)
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256:
{% set version = '4.0.56' %} ... sha256: 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d ... |
Set environment:
source /cds/sw/ds/ana/conda1/inst/etc/profile.d/conda.sh # OR conda deactivate after previous conda_build
conda activate conda_build_py3
Execute command:
conda build -c lcls-i -c conda-forge recipe
> /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2
xorg-xf86vidmodeproto: 2.3.1-h7f98852_1002 conda-forge xorg-xproto: 7.0.31-h7f98852_1007 conda-forge xz: 5.2.6-h166bdaf_0 conda-forge zeromq: 4.3.5-h59595ed_0 conda-forge zipp: 3.17.0-pyhd8ed1ab_0 conda-forge zlib: 1.2.13-hd590300_5 conda-forge zstd: 1.5.5-hfc55251_0 conda-forge Preparing transaction: ...working... done Verifying transaction: ...working... done Executing transaction: ...working... done export PREFIX=/cds/home/p/psreldev/conda-bld/psana_1700187012881/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pl export SRC_DIR=/cds/home/p/psreldev/conda-bld/psana_1700187012881/test_tmp + psana -h Usage: psana [options] [dataset ...] Available options: {-h|-?|--help } print help message {-v|--verbose } (incr) verbose output, multiple allowed (initial: 0) {-q|--quiet } (incr) quieter output, multiple allowed (initial: 2) {-b|--calib-dir } path calibration directory name, may include {exp} and {instr}, if left empty then do not do calibrations (default: "") {-c|--config } path configuration file, by default use psana.cfg if it exists (default: "") {-e|--experiment } string experiment name, format: XPP:xpp12311 or xpp12311, by default guess it from data (default: "") {-j|--job-name } string job name, default is to generate from input file names (default: "") {-m|--module } name module name, more than one possible {-n|--num-events } number maximum number of events to process, 0 means all (default: 0) {-s|--skip-events} number number of events to skip (default: 0) {-p|--num-cpu } number number greater than 0 enables multi-processing (default: 0) {-o|--option } string configuration options, format: module.option[=value] Positional parameters: dataset - input dataset specification (list of file names or exp=cxi12345:run=123:...) + exit 0 Resource usage statistics from testing psana: Process count: 1 CPU time: Sys=0:00:00.1, User=- Memory: 1.4M Disk usage: 12B Time elapsed: 0:00:02.4 TEST END: /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2 Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' INFO:conda_build.utils:Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' INFO conda_build.utils:shutil_move_more_retrying(2075): Renaming work directory '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work' to '/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop' shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) INFO:conda_build.utils:shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) INFO conda_build.utils:shutil_move_more_retrying(2082): shutil.move(work)=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work, dest=/cds/home/p/psreldev/conda-bld/psana_1700187012881/work_moved_psana-4.0.56-py39hb869b97_1_linux-64_main_build_loop) # Automatic uploading is disabled # If you want to upload package(s) to anaconda.org later, type: anaconda upload /cds/home/p/psreldev/conda-bld/linux-64/psana-4.0.56-py39hb869b97_1.tar.bz2 # To have conda build upload to anaconda.org automatically, use # $ conda config --set anaconda_upload yes anaconda_upload is not set. Not uploading wheels: [] #################################################################################### Resource usage summary: Total time: 1:22:33.6 CPU usage: sys=0:03:04.5, user=0:27:14.5 Maximum memory usage observed: 1.6G Total disk usage observed (not including envs): 285.6M #################################################################################### Source and build intermediates have been left in /cds/home/p/psreldev/conda-bld. There are currently 1 accumulated. To remove them, you can run the ```conda build purge``` command (conda_build) [psreldev@psbuild-rhel7-01 psana1-feedstock]$ |
anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ pwd /sdf/home/p/psreldev/git/psana1-feedstock (conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2 Using Anaconda API: https://api.anaconda.org Using "lcls-i" as upload username Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" Detecting file type... File type is "Conda" Extracting conda attributes for upload Creating package "psana" Creating release "4.0.57" The action you are performing requires authentication, please sign in: Using Anaconda API: https://api.anaconda.org Username: dubrovin dubrovin's Password: login successful Using Anaconda API: https://api.anaconda.org Using "lcls-i" as upload username Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" Detecting file type... File type is "Conda" Extracting conda attributes for upload Creating package "psana" Creating release "4.0.57" Uploading file "lcls-i/psana/4.0.57/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" 15.4MB [00:01, 12.6MB/s] Upload complete conda located at: https://anaconda.org/lcls-i/psana (conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ |
goal: create a real 4.0.57
conda create --name psana-conda-4.0.57 --clone psana-conda-4.0.56
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ conda create --name ana-4.0.57-py3 --clone ana-4.0.55-py3 Source: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.55-py3 Destination: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.57-py3 Packages: 488 Files: 11 Downloading and Extracting Packages: font-ttf-dejavu-sans | ########################################################################################################################################################################### | 100%^ Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: | For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default. To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before launching your MPI processes. Equivalently, you can set the MCA parameter in the command line: mpiexec --mca opal_cuda_support 1 ... In addition, the UCX support is also built but disabled by default. To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes. Equivalently, you can set the MCA parameters in the command line: mpiexec --mca pml ucx --mca osc ucx ... Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX. Please consult UCX's documentation for detail. done # # To activate this environment, use # # $ conda activate ana-4.0.57-py3 # # To deactivate an active environment, use # # $ conda deactivate (conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ |
conda deactivate
conda activate psana-conda-4.0.57
conda install -c lcls-i -c conda-forge psana=4.0.57 # Chris? --experimental-solver=libmamba
have to do this on both old-psana and new-s3df as psreldev
See details in ~/.bash_history or ~/.k5login
cd /cds/sw/ds/ana/conda1/
if ./manage is not available
git clone git@github.com:slaclab/anarel-manage.git manage
cd manage
update it if necessary
git pull --rebase
conda create -n conda_build_py3 python=3.9 anaconda
conda env remove --name conda_build_py3
see /cds/sw/ds/ana/conda1/inst/envs/ OR conda info --envs
ssh psbuild-rhel7-01 -l psreldev
cd git/psana1-feedstock/
sha256sum /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
> 443440bf7a7fdbcdf89314b68c07f97f5a89dff38d6da98685e74f0671fac37d
~psreldev/git/psana1-feedstock/recipe/meta.yaml
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
Set environment (chris):
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh. # Chris: conda2 because it needs py3, but conda1 also works with py3...
conda deactivate
conda activate conda_build
OR:
source /cds/sw/ds/ana/conda2/manage/bin/psconda.sh
conda create -n conda_build_py3 python conda-build anaconda-client
conda activate conda_build_py3
ssh psbuild-rhel7-01 -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
cp /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz /reg/g/psdm/web/swdoc/tutorials/
Issue: PermissionError: [Errno 13] Permission denied: '/cds/sw/ds/ana/conda2/inst/envs/conda_build/pkgs/libselinux-cos6-x86_64-2.0.94-h9b0a68f_1105'
Chris: cp ~cpo/.condarc ~/. # if ~/.condarc is not available
Valerio: conda create -n local_conda_build python conda-build conda-validate anaconda-client
tar cfvz /cds/sw/ds/ana/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz psana-conda-4.0.56
IT IS ALREADY DONE ONCE
cd .../con-build OR cd /sdf/group/lcls/ds/ana/sw/conda1/
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
git clone git@github.com:slaclab/anarel-manage.git
ema ~/.gitconfig
\[http\] - remove
> s3dfnx ends up as dubrovin@sdflogin002
> psana ... because /afs/slac/g/pcds/svn/pdsdata is seen from psana nodes only... (Wilko)
cd <path-to>/anarel-manage
> s3dfnx ends up as dubrovin@sdflogin002
ssh psana -l psreldev
cd /sdf/group/lcls/ds/ana/sw/conda1/manage
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate conda_build
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.56 --basedir `pwd`
Command bin/ana-rel-admin creates file
/sdf/group/lcls/ds/ana/sw/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
accessible in ~psreldev/git/psana1-feedstock/recipe/meta.yaml through url:
file:///sdf/group/lcls/ds/ana/sw/conda1/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
Valerio generated new conda1-v3
source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh
conda deactivate
conda activate conda_build
(conda_build2) [psreldev@sdfiana002 manage]$ conda --version
conda 23.10.0
source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh
content changes for conda1-v3
- eval "$(/sdf/group/lcls/ds/ana/sw/conda1/inst/bin/conda shell.bash hook)"
+ eval "$(/sdf/group/lcls/ds/ana/sw/conda1-v3/inst/bin/conda shell.bash hook)"
...
- eval "$(/cds/sw/ds/ana/conda1-v2/inst/bin/conda shell.bash hook)"
+ eval "$(/cds/sw/ds/ana/conda1-v3/inst/bin/conda shell.bash hook)"
~psreldev/git/psana1-feedstock/recipe/meta.yaml
accessible in ~psreldev/git/psana1-feedstock/recipe/meta.yaml (ADD -v3 !!!):
file:///sdf/group/lcls/ds/ana/sw/conda1-v3/manage/downloads/anarel/psana-conda-4.0.56.tar.gz
Everything works on s3df psana node under login psreldev because it needs some disk space... Other than that it can be built under any. other user.
IT IS ALREADY DONE ONCE for psreldev
> s3dflogin
ssh psana -l psreldev
cd git
rm -rf psana1-feedstock/
git clone https://github.com/slac-lcls/psana1-feedstock
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate # > base
conda activate base
conda install conda-build
conda install conda-verify # also works in the base
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate conda_build
cd ~/git/psana1-feedstock/
Update in ~psreldev/git/psana1-feedstock/recipe/meta.yaml
fields for set version and sha256
conda build -c lcls-i -c conda-forge recipe
svn co file:///afs/slac/g/pcds/svn/pdsdata/tags/V09-04-00 extpkgs/pdsdata
svn co file:///afs/slac/g/pcds/svn/psalg/tags/V01-00-12 extpkgs/psalg
created drepositories on git@github.com:lcls-psana/
git clone git@github.com:lcls-psana/pdsdata.git
git clone git@github.com:lcls-psana/psalg.git
s3dfnx - nx - server on s3df connects to dubrovin@sdflogin002
. /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda create -n conda_build_svn svn anaconda-client
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda activate conda_build_svn
git tag -d V10-00-01 # removes tag on local git
git push --delete origin V10-00-01 # removes tag on github
> s3dflogin
ssh psana -l psreldev
source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh # conda1-v3 is a latest version of conda 23.10.0
conda deactivate
conda activate conda_build
cd /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/ # if needed git clone git@github.com:slaclab/anarel-manage.git manage
bin/ana-rel-admin --force --cmd psana-conda-src --name 4.0.58 --basedir `pwd`
cd ~psreldev/git/psana1-feedstock/ # if needed git clone git@github.com:slac-lcls/psana1-feedstock.git
Update in recipe/meta.yaml fields for set version and sha256
conda build -c lcls-i -c conda-forge recipe
In case of problem with tests look at log file like
/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/psana_<13-digit-build number>/test_tmp/work/<log-file-name>
anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ pwd /sdf/home/p/psreldev/git/psana1-feedstock (conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2 Using Anaconda API: https://api.anaconda.org Using "lcls-i" as upload username Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" Detecting file type... File type is "Conda" Extracting conda attributes for upload Creating package "psana" Creating release "4.0.57" The action you are performing requires authentication, please sign in: Using Anaconda API: https://api.anaconda.org Username: dubrovin dubrovin's Password: login successful Using Anaconda API: https://api.anaconda.org Using "lcls-i" as upload username Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" Detecting file type... File type is "Conda" Extracting conda attributes for upload Creating package "psana" Creating release "4.0.57" Uploading file "lcls-i/psana/4.0.57/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2" 15.4MB [00:01, 12.6MB/s] Upload complete conda located at: https://anaconda.org/lcls-i/psana (conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ |
conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3
(base) [psreldev@sdfiana002 psana1-feedstock]$ conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3 Retrieving notices: ...working... done Source: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.57-py3 Destination: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3 Packages: 488 Files: 11 Downloading and Extracting Packages: Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: / For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default. To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before launching your MPI processes. Equivalently, you can set the MCA parameter in the command line: mpiexec --mca opal_cuda_support 1 ... In addition, the UCX support is also built but disabled by default. To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes. Equivalently, you can set the MCA parameters in the command line: mpiexec --mca pml ucx --mca osc ucx ... Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX. Please consult UCX's documentation for detail. done # # To activate this environment, use # # $ conda activate ana-4.0.58-py3 # # To deactivate an active environment, use # # $ conda deactivate (base) [psreldev@sdfiana002 psana1-feedstock]$ |
conda deactivate
conda activate ana-4.0.58-py3
conda install -c lcls-i -c conda-forge psana=4.0.58
(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ conda install -c lcls-i -c conda-forge psana=4.0.58 Channels: - lcls-i - conda-forge - defaults - lcls-ii - cogsci Platform: linux-64 Collecting package metadata (repodata.json): done Solving environment: done ## Package Plan ## environment location: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3 added / updated specs: - psana=4.0.58 The following packages will be downloaded: package | build ---------------------------|----------------- psana-4.0.58 | py39hed0727e_1 15.4 MB lcls-i ------------------------------------------------------------ Total: 15.4 MB The following packages will be UPDATED: psana 4.0.57-py39hed0727e_1 --> 4.0.58-py39hed0727e_1 Proceed ([y]/n)? y Downloading and Extracting Packages: Preparing transaction: done Verifying transaction: done Executing transaction: done (ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ |
Info from Valerio
activate ana-4.0.58-py3 on S3DF,
>s3dflogin
ssh psana -l psreldev
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda env list
conda activate ana-4.0.58-py3
then:
conda list --explicit > ana-4.0.58-py3.txt
copy ana-4.0.58-py3.txt to psreldev on PCDS - do this trick due to permissions:
>s3dflogin
cp ~psreldev/ana-4.0.61-py3.txt .
> pslogin
ssh psbuild-rhel7-01 -l psreldev
scp dubrovin@s3dflogin.slac.stanford.edu:ana-4.0.61-py3.txt . # complete copy of ana-4.0.61-py3.txt
source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt
(ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$ conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt Downloading and Extracting Packages plotly-5.13.1 | ############################################################################################################################################################## | 100% gdk-pixbuf-2.42.10 | ############################################################################################################################################################## | 100% graphviz-7.1.0 | ############################################################################################################################################################## | 100% ipykernel-6.20.2 | ############################################################################################################################################################## | 100% psana-4.0.58 | ############################################################################################################################################################## | 100% Preparing transaction: done Verifying transaction: done Executing transaction: - For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default. To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before launching your MPI processes. Equivalently, you can set the MCA parameter in the command line: mpiexec --mca opal_cuda_support 1 ... In addition, the UCX support is also built but disabled by default. To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes. Equivalently, you can set the MCA parameters in the command line: mpiexec --mca pml ucx --mca osc ucx ... Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX. Please consult UCX's documentation for detail. done (ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$ |
edit and change release name in
pcds: /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
s3df: /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh > conda1-v3
Info from Chris:
ssh to pcds node > pslogin
ssh psbuild-rhel7-01 -l psreldev
Update version in TWO lines:
/cds/group/psdm/sw/conda/jhub_config/prod-rhel7/kernels/ana1-current-py3/kernel.json