...

Option 2 (preferred): create your own conda environment. Follow the SDF docs to use ATLAS group installation of conda.
There is also a slight hiccup with permissions in the folder /sdf/group/atlas/sw/conda/pkgs, which one can sidestep by specifying their own folder for saving packages (in GPFS data space).
The TLDR is:

export PATH="/sdf/group/atlas/sw/conda/bin:$PATH" 
conda init # the previous will be added to your bashrc file
# Add the following lines to ~/.condarc file (create default file with conda config)
pkgs_dirs:
  - /gpfs/slac/atlas/fs1/d/<user>/conda_envs

/<user>/conda_envs 
envs_dirs:
  - /gpfs/slac/atlas/fs1/d/<user>/conda_envs
channels:
  - defaults
  - conda-forge

This will allow you to also create environments saved in GPFS rather than the login node, which fills up quickly, using –prefix /gpfs/slac/atlas/fs1/d/<user>/conda_envs/env_name .

conda init # the previous will be added to your bashrc file 
conda env create -f bjr_v01.yaml # for example(bjr_v01) conda install jupyter

...

/gpfs/slac/atlas/fs1/d/bbullard/conda_env_yaml/
├── puma.yaml
├── salt.yaml
└── upp.yaml

Add the following to your .bashrc script as well

export PATH="/sdf/group/atlas/sw/conda/bin:$PATH"
export TMPDIR="${SCRATCH}"

...

Producing H5 samples with Training Dataset Dumper

...

If using the latest main branch from Dec 11 2023, after SALT model refactoring has occurred, there are some issues with CUDA version compatibility with SDF.
PyTorch v2.0.1 used in SALT requires newer CUDA version than is available (11.4). One may need to do the following:

conda config --append channels conda-forge # This is useful elsewhere too
conda uninstall pytorch
conda install pytorch torchvision torchaudio cudatoolkit=11.4 -c pytorch

...