...

1. GitHub Access Token

   
   A new set of python scripts has been developed to automate the creation of LCLS-II conda environment. They can be found here:
   

   /cds/sw/ds/ana/conda2/manage/buildenv

   The scripts need a valid personal GitHub token to be used (unfortunately, GitHub caps the number of API requests that can be done without a token). The token can be created on GitHub, after logging in:

   https://github.com/settings/tokens

   The token must be exported and made available as an environment variable called GITHUB_ACCESS_TOKEN.  Technically it is only needed by step (4) below, but the way the scripts are written it should be set for all steps.

   For an example, please see ~valmar's bashrc file (also in psrel bashrc).
   
   

2. Package Version File

   
   (this file-editing should be run as user "psrel")
   
   In order to generate packages, a YAML file listing the required version of each package must be created. For a file with the package versions in the latest environments, see:
   /cds/sw/ds/ana/conda2/manage/buildenv/table.yaml
   

   A few lines as an example:

   ami: 2.4.7
   amityping: 1.1.7
   cameralink-gateway: 7.6.2
   epix: 0.0.3
   lcls2-pgp-pcie-apps: 2.2.0
   lcls2_timetool: 3.3.0
   epix-hr-single-10k: 3.1.1
   lcls2-epix-hr-pcie: 1.2.0
   ....

3. Preparing source tarballs

   
   (this step should be run as a normal user like valmar, cpo,...)
   
   Before building the feedstocks, source tarballs must be created for the rogue-related packages (which are private repos, so need source .tgz files generated) by running the prepare_source.py script.
   
   The script must be run using the conda_build environment (conda activate conda_build) It must also be run as a normal user because psrel cannot write to the /reg/g/psdm/web/swdoc/tutorials/.
   The script takes the package version file as an argument:
   

   python /cds/sw/ds/ana/conda2/manage/buildenv/prepare_source.py --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml

   
   The source tarballs are automatically generated and copied to /reg/g/psdm/web/swdoc/tutorials/
   
   PS: The script will clone the required repostiories in the current working directory!!!! Working in a temporary directory that can be later deleted is strongly advised
   
   

4. Generating packages

   
   (this step should be run as a normal user like valmar, cpo,...)
   
   The packages can now be built using the build environment script, again in the conda_build environment (conda activate conda_build):
   

   python /cds/sw/ds/ana/conda2/manage/buildenv/build_environment.py --generate-packages --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml

   
   The script will build the packages wave-by wave. For each wave, the script will clone the feedstocks one by one, automatically make the necessary changes to the recipes, run "conda smithy" and push the changes to the git repositories, triggering the building of the packages.  The individual builds can be seen at URLs like https://github.com/slac-lcls/libnl3-feedstock/actions
   
   The script will then check with GitHub every thirty seconds, and report the status of the build process ("Not started yet", "running", "success" or "failed"). It will waint until all builds have finished and are either in a "success" or "failed" state. If no build has failed, the script will then proceed to the next package "wave". Otherwise it will exit.
   
   Instead of going through all the waves, one can start from a certain wave (using the --start-from-wave option) and/or stop at a certain wave (using the --stop-at-wave option)
   
   PS: The script will clone the required repostiories in the current working directory!!!! Working in a temporary directory that can be later deleted is strongly advised
   
   

5. Building environments

   
   (this step should be run as user psrel)
   
   The production and development environments can be created using the normal "conda env create" commands (see above), or using the build_environment script:
   

   (from /cds/sw/ds/ana/conda2/manage directory)
   (NOTE: this command produced the env in a non-standard location, cpo used "conda env create --name ps-4.4.1 -f prod_create.yaml" instead)
   python buildenv/build_environment.py --build-environment --environment-name=ps-4.4.0 --environment-file=prod_create.yaml

   
   The environments must be created using the psrel user, because only the psrel user can write to the environment directories
   
   This step and the previous one can be combined in a single command for convenience:
   

   python build_environment.py --generate-packages --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml --build-environment --environment-name=ps-4.4.0 --environment-file=prod_create.yaml

LCLS2 Pinned Packages Approach (Current as of 10/9/23)

Building A Package

Need to create a feedstock for every new package.  LCLS2 feedstock packages are in (for example) https://github.com/slac-lcls/epix-feedstock.

...