Page History

...

1. GitHub Access Token

   
   A new set of python scripts has been developed to automate the creation of LCLS-II conda environment. They can be found here:
   

   /cds/sw/ds/ana/conda2/manage/buildenv

   The scripts need a valid personal GitHub token to be used (unfortunately, GitHub caps the number of API requests that can be done without a token). The token can be created on GitHub, after logging in:

   https://github.com/settings/tokens

   The token must be exported and made available as an environment variable called GITHUB_ACCESS_TOKEN

   For an example, please see ~valmar's bashrc file
   
   

2. Package Version File

   
   In order to generate packages, a YAML file listing the reuired required version of each package must be created. For a file with the package versions in the latest environments, see:
   /cds/sw/ds/ana/conda2/manage/buildenv/table.yaml

   A few lines as an example:

   ami: 2.4.7
   amityping: 1.1.7
   cameralink-gateway: 7.6.2
   epix: 0.0.3
   lcls2-pgp-pcie-apps: 2.2.0
   lcls2_timetool: 3.3.0
   epix-hr-single-10k: 3.1.1
   lcls2-epix-hr-pcie: 1.2.0
   ....

3. Preparing source tarballs

   
   Before building the feedstocks, source tarballs must be created for the rogue-related packages (which are private repos, so need source .tgz files generated) by running the prepare_source.py script.
   
   The script must be run using the conda_build environment (conda activate conda_build) It must also be run as a normal user because psrel cannot write to the /reg/g/psdm/web/swdoc/tutorials/.
   The script takes the package version file as an argument:
   

   python /cds/sw/ds/ana/conda2/manage/buildenv/prepare_source.py --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml

   
   The source tarballs are automatically generated and copied to /reg/g/psdm/web/swdoc/tutorials/
   
   PS: The script will clone the required repostiories in the current working directory!!!! Working in a temporary directory that can be later deleted is strongly advised
   
   

4. Generating packages

   
   The packages can now be built using the build environment script, again in the conda_build environment (conda activate conda_build):
   

   python /cds/sw/ds/ana/conda2/manage/buildenv/build_environment.py --generate-packages --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml

   
   The script will build the packages wave-by wave. For each wave, the script will clone the feedstocks one by one, automatically make the necessary changes to the recipes automatically, run "conda smithy" and push the changes to the git repositories, triggering the building of the packages.  The individual builds can be seen at URLs like https://github.com/slac-lcls/libnl3-feedstock/actions
   
   The script will then check with GitHub every thirty seconds, and report the status of the build process ("Not started yet", "running", "success" or "failed"). It will waint until all builds have finished and are either in a "success" or "failed" state. If no build has failed, the script will then proceed to the next package "wave". Otherwise it will exit.
   
   Instead of going through all the waves, one can start from a certain wave (using the --start-from-wave option) and/or stop at a certain wave (using the --stop-at-wave option)
   
   PS: The script will clone the required repostiories in the current working directory!!!! Working in a temporary directory that can be later deleted is strongly advised
   
   

5. Building environments

   
   The production and development environments can be created using the normal "conda env create" commands (see above), or using the build_environment script:
   

   python build_environment.py --build-environment --environment-name=ps-4.4.0 --environment-file=prod_create.yaml

   
   The environments must be created using the psrel user, because only the psrel user can write to the environment directories
   
   This step and the previous one can be combined in a single command for convenience:
   

   python build_environment.py --generate-packages --package-version-file=/cds/sw/ds/ana/conda2/manage/buildenv/table.yaml --build-environment --environment-name=ps-4.4.0 --environment-file=prod_create.yaml