...

source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh
conda deactivate
conda activate conda_build2build

(conda_build2) [psreldev@sdfiana002 manage]$ conda --version
conda 23.10.0

...

git push --delete origin V10-00-01 # removes tag on github

...

List of commands to build psana1 release on s3df

Login and set environment

...

source /sdf/group/lcls/ds/ana/sw/conda1-v3/manage/bin/psconda.sh # conda1-v3 is a latest version of conda 23.10.0
conda deactivate
conda activate conda_build2build

Generate .tar.gz file with source code

...

conda build -c lcls-i -c conda-forge recipe

Debugging

In case of problem with tests look at log file like

/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/psana_<13-digit-build number>/test_tmp/work/<log-file-name>

Upload .tar.gz file with release to anaconda lcls-i channel

anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2

Create new environment

conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3

Code Block

title	response on conda createanaconda upload
collapse	true

(baseconda_build2) [psreldev@sdfiana002psreldev@sdfiana001 psana1-feedstock]$ conda create --name ana-4.0.58-py3 --clone anapwd
/sdf/home/p/psreldev/git/psana1-feedstock
(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$ anaconda upload -u lcls-i /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py3
Retrieving notices: ...working... done
Source:      py39hed0727e_1.tar.bz2
Using Anaconda API: https://api.anaconda.org
Using "lcls-i" as upload username
Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/anaconda_build2/conda-bld/linux-64/psana-4.0.57-py3
Destination: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3
Packages: 488
Files: 11

Downloading and Extracting Packages:

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: /  
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
 
In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.
 
done
#
# To activate this environment, use
#
#     $ conda activate ana-4.0.58-py3
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) [psreldev@sdfiana002 psana1-feedstock]$

conda deactivate
conda activate ana-4.0.58-py3
conda install -c lcls-i -c conda-forge psana=4.0.58

Code Block

title	response on conda install
collapse	true

(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ conda install -c lcls-i -c conda-forge psana=4.0.58
Channels:
 - lcls-i
 - conda-forge
 - defaults
 - lcls-ii
 - cogsci
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3

  added / updated specs:
    - psana=4.0.58


The following packages will be downloaded:

    packagepy39hed0727e_1.tar.bz2"
Detecting file type...
File type is "Conda"
Extracting conda attributes for upload
Creating package "psana"
Creating release "4.0.57"
The action you are performing requires authentication, please sign in:
Using Anaconda API: https://api.anaconda.org
Username: dubrovin
dubrovin's Password: 
login successful
Using Anaconda API: https://api.anaconda.org
Using "lcls-i" as upload username
Processing "/sdf/group/lcls/ds/ana/sw/conda1/inst/envs/conda_build2/conda-bld/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2"
Detecting file type...
File type is "Conda"
Extracting conda attributes for upload
Creating package "psana"
Creating release "4.0.57"
Uploading file "lcls-i/psana/4.0.57/linux-64/psana-4.0.57-py39hed0727e_1.tar.bz2"
15.4MB [00:01, 12.6MB/s]                                                                                                                                                                         |         
Upload complete

conda located buildat:
  https://anaconda.org/lcls-i/psana

(conda_build2) [psreldev@sdfiana001 psana1-feedstock]$

Create new environment

conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3

Code Block

title	response on conda create
collapse	true

(base) [psreldev@sdfiana002 psana1-feedstock]$ conda create --name ana-4.0.58-py3 --clone ana-4.0.57-py3
Retrieving notices: ...working... done
Source:      /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.57-py3
Destination: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3
Packages: 488
Files: 11

Downloading and Extracting Packages:

Downloading and Extracting Packages:

Preparing transaction: done
Verifying transaction: done
Executing transaction: /  
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
 
In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.
 
done
#
# To activate this environment, use
#
#     $ conda activate ana-4.0.58-py3
#
# To deactivate an active environment, use
#
#     $ conda deactivate

(base) [psreldev@sdfiana002 psana1-feedstock]$

conda deactivate
conda activate ana-4.0.58-py3
conda install -c lcls-i -c conda-forge psana=4.0.58

Code Block

title	response on conda install
collapse	true

(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$ conda install -c lcls-i -c conda-forge psana=4.0.58
Channels:
 - lcls-i
 - conda-forge
 - defaults
 - lcls-ii
 - cogsci
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.58-py3

  added / updated specs:
    - psana=4.0.58


The following packages will be downloaded:

    package---------------------------|-----------------
    psana-4.0.58               |   py39hed0727e_1        15.4 MB  lcls-i
    ------------------------------------------------------------
                                           Total:        15.4 MB

The following packages will be UPDATED:

  psana                               4.0.57-py39hed0727e_1 --> 4.0.58-py39hed0727e_1 


Proceed ([y]/n)? y


Downloading and Extracting Packages:
                                                                                                                  |            build
              ---------------------------|-----------------
    psana-4.0.58               |   py39hed0727e_1        15.4 MB  lcls-i
    ------------------------------------------------------------
                   
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(ana-                        Total:        15.4 MB

The following packages will be UPDATED:

  psana                               4.0.57-py39hed0727e_1 --> 4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$py39hed0727e_1 


Proceed ([y]/n)? y


Downloading and Extracting Packages:
                                                                                                                                                                                                    
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
(ana-4.0.58-py3) [psreldev@sdfiana002 psana1-feedstock]$

Create the same release on pcds

Info from Valerio

activate ana-4.0.58-py3 on S3DF,

>s3dflogin

ssh psana -l psreldev

source /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh

conda deactivate

conda env list

conda activate ana-4.0.58-py3

then:

conda list --explicit > ana-4.0.58-py3.txt

copy ana-4.0.58-py3.txt to psreldev on PCDS - do this trick due to permissions:

>s3dflogin

cp ~psreldev/ana-4.0.61-py3.txt .

> pslogin

ssh psbuild-rhel7-01 -l psreldev

scp dubrovin@s3dflogin.slac.stanford.edu:ana-4.0.61-py3.txt . # complete copy of ana-4.0.61-py3.txt

source /cds/sw/ds/ana/conda1/manage/bin/psconda.sh

conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt

Code Block

title	response on conda create
collapse	true

(ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$ conda create -n ana-4.0.58-py3 --file ana-4.0.58-py3.txt

Downloading and Extracting Packages
plotly-5.13.1        | ############################################################################################################################################################## | 100% 
gdk-pixbuf-2.42.10   | ############################################################################################################################################################## | 100% 
graphviz-7.1.0       | ############################################################################################################################################################## | 100% 
ipykernel-6.20.2     | ############################################################################################################################################################## | 100% 
psana-4.0.58         | ############################################################################################################################################################## | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: -  
For Linux 64, Open MPI is built with CUDA awareness but this support is disabled by default.
To enable it, please set the environment variable OMPI_MCA_opal_cuda_support=true before
launching your MPI processes. Equivalently, you can set the MCA parameter in the command line:
mpiexec --mca opal_cuda_support 1 ...
 
In addition, the UCX support is also built but disabled by default.
To enable it, first install UCX (conda install -c conda-forge ucx). Then, set the environment
variables OMPI_MCA_pml="ucx" OMPI_MCA_osc="ucx" before launching your MPI processes.
Equivalently, you can set the MCA parameters in the command line:
mpiexec --mca pml ucx --mca osc ucx ...
Note that you might also need to set UCX_MEMTYPE_CACHE=n for CUDA awareness via UCX.
Please consult UCX's documentation for detail.
 

done
(ana-4.0.54-py3) [psreldev@psbuild-rhel7-01 ~]$

Install new release on pcds:

conda install --experimental-solver=libmamba -c lcls-i -c conda-forge psana=4.0.58 # RECOMMENDED

conda install -c lcls-i -c conda-forge psana=4.0.58 # WORKING

edit and change release name in

pcds: /cds/sw/ds/ana/conda1/manage/bin/psconda.sh

s3df: /sdf/group/lcls/ds/ana/sw/conda1/manage/bin/psconda.sh > conda1-v3

Update release for Jupyter Notebook

Info from Chris:

ssh to pcds node > pslogin

ssh psbuild-rhel7-01 -l psreldev

Update version in TWO lines:

/cds/group/psdm/sw/conda/jhub_config/prod-rhel7/kernels/ana1-current-py3/kernel.json

Tags of packages included in the release

on pcds: /cds/sw/ds/ana/conda1/inst/envs/<environment>/data/anarelinfo/psana-conda-tags

on s3df: /sdf/group/lcls/ds/ana/sw/conda1/inst/envs/ana-4.0.62-py3/data/anarelinfo/psana-conda-tags

Detector conda_branch=False repo=psdm subdir=Non tag=V02-00-84 ?????

References

Building Conda Packages And Releases
conda-cheatsheet.pdf
https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#cloning-an-environment
/reg/g/psdm/etc/psconda.sh -> /cds/sw/ds/ana/conda1/manage/bin/psconda.sh
Running at S3DF
Convert svn repository to git
s3df-dynamic-sites-and-web-applications

...

Page tree

Versions Compared

Old Version 48

New Version Current

Key

List of commands to build psana1 release on s3df

Login and set environment

Generate .tar.gz file with source code

Debugging

Upload .tar.gz file with release to anaconda lcls-i channel

Create new environment

Create new environment

Create the same release on pcds

Update release for Jupyter Notebook

Tags of packages included in the release

References

Page tree

Page History

Versions Compared

Old Version 48

New Version Current

Key

List of commands to build psana1 release on s3df

Login and set environment

Generate .tar.gz file with source code

Debugging

Upload .tar.gz file with release to anaconda lcls-i channel

Create new environment

Create new environment

Create the same release on pcds

Update release for Jupyter Notebook

Tags of packages included in the release

References