Page History
...
source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh
conda deactivate
conda activate conda_build2build
(conda_build2) [psreldev@sdfiana002 manage]$ conda --version
conda 23.10.0
...
git push --delete origin V10-00-01 # removes tag on github
...
List of commands to build psana1 release on s3df
Login and set environment
...
source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh # conda1-v3 is a latest version of conda 23.10.0
conda deactivate
conda activate conda_build2build
Generate .tar.gz file with source code
...
conda build -c lcls-i -c conda-forge recipe
Debugging
In case of problem with tests look at log file like
/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/psana_<13-digit-build number>/test_tmp/work/<log-file-name>
Upload .tar.gz file with release to anaconda lcls-i channel
anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2
Create new environment
conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3
Code Block | ||||
---|---|---|---|---|
| ||||
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ pwd
/sdf/home/p/psreldev/git/psana1-feedstock
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "lcls-i" as upload username
Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2"
Detecting file type...
File type is "Conda"
Extracting conda attributes for upload
Creating package "psana"
Creating release "4.0.57"
The action you are performing requires authentication, please sign in:
Using Anaconda API: https://api.anaconda.org
Username: dubrovin
dubrovin's Password:
login successful
Using Anaconda API: https://api.anaconda.org
Using "lcls-i" as upload username
Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2"
Detecting file type...
File type is "Conda"
Extracting conda attributes for upload
Creating package "psana"
Creating release "4.0.57"
Uploading file "lcls-i/psana/4.0.57/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2"
15.4MB [00:01, 12.6MB/s]
Upload complete
conda located at:
https://anaconda.org/lcls-i/psana
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ |
Create new environment
conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3
Code Block | ||||
---|---|---|---|---|
| ||||
(base) [psreldev@sdfiana002 psana1-feedstock]$ conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3
Retrieving notices: ...working... done
Source: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.57-py3
Destination: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3
Packages: 488
Files: 11
Downloading and Extracting Packages:
Downloading and Extracting Packages:
Preparing transaction: done
Verifying transaction: done
Executing transaction: /
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.
done
#
# To activate this environment, use
#
# $ conda activate ana-4.0.58-py3
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) [psreldev@sdfiana002 psana1-feedstock]$ |
conda deactivate
conda activate ana-4.0.58-py3
conda install -c lcls-i -c conda-forge psana=4.0.58
Code Block | ||||
---|---|---|---|---|
| ||||
(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ conda install -c lcls-i -c conda-forge psana=4.0.58
Channels:
- lcls-i
- conda-forge
- defaults
- lcls-ii
- cogsci
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3
added / updated specs:
- psana=4.0.58
The following packages will be downloaded:
package | build
---------------------------|-----------------
psana-4.0.58 | py39hed0727e_1 15.4 MB lcls-i
------------------------------------------------------------
Total: 15.4 MB
The following packages will be UPDATED:
psana 4.0.57-py39hed0727e_1 --> 4.0.58-py39hed0727e_1
Proceed ([y]/n)? y
Downloading and Extracting Packages:
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ |
Create the same release on pcds
Info from Valerio
activate ana-4.0.58-py3 on S3DF,
>s3dflogin
ssh psana -l psreldev
source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh
conda deactivate
conda env list
conda activate ana-4.0.58-py3
then:
conda list --explicit > ana-4.0.58-py3.txt
copy ana-4.0.58-py3.txt to psreldev on PCDS - do this trick due to permissions:
>s3dflogin
cp ~psreldev/ana-4.0.61-py3.txt .
> pslogin
ssh psbuild-rhel7-01 -l psreldev
scp dubrovin@s3dflogin.slac.stanford.edu:ana-4.0.61-py3.txt . # complete copy of ana-4.0.61-py3.txt
source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt
Code Block | ||||
---|---|---|---|---|
| ||||
(ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$ conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt
Downloading and Extracting Packages
plotly-5.13.1 | ############################################################################################################################################################## | 100%
gdk-pixbuf-2.42.10 | ############################################################################################################################################################## | 100%
graphviz-7.1.0 | ############################################################################################################################################################## | 100%
ipykernel-6.20.2 | ############################################################################################################################################################## | 100%
psana-4.0.58 | ############################################################################################################################################################## | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: -
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.
done
(ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$ |
Install new release on pcds:
conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.58 # RECOMMENDED
conda install -c lcls-i -c conda-forge psana=4.0.58 # WORKING
edit and change release name in
pcds: /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
s3df: /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh > conda1-v3
Update release for Jupyter Notebook
Info from Chris:
ssh to pcds node > pslogin
ssh psbuild-rhel7-01 -l psreldev
Update version in TWO lines:
/cds/group/psdm/sw/conda/jhub_config/prod-rhel7/kernels/ana1-current-py3/kernel.json
Tags of packages included in the release
on pcds: /cds/sw/ds/ana/conda1/inst/envs/<environment>/data/anarelinfo/psana-conda-tags
on s3df: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.62-py3/data/anarelinfo/psana-conda-tags
Detector conda_branch=False repo=psdm subdir=Non tag=V02-00-84 ?????conda deactivate
conda activate ana-4.0.58-py3
conda install -c lcls-i -c conda-forge psana=4.0.58
References
- Building Conda Packages And Releases
- conda-cheatsheet.pdf
- https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
- /reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
- Running at S3DF
- Convert svn repository to git
- s3df-dynamic-sites-and-web-applications
...