Introduction
This tutorial will cover how to run a parameter scan on FACET-II using quasi-random number selection of parameters through parallel computing on NERSC (National Energy Research Scientific Computing Center). Currently, NERSC has problems with running MPI application from Docker containers, so setting up a virtual environment is a temporary and effective alternative. These parameter scan scripts are created by Eric Cropp.
NERSC Account Setup
If you have not already done so, request a NERSC account via Iris.
PITFALL Make sure that you have a project ID from your supervisor before trying to request an account. It usually starts with the letter 'm' followed by four digits.
After your request for a NERSC account is approved, set up multi-factor authentication (MFA). This is required to access NERSC's systems via SSH.
Setup Parameter Scan Locally
Go to Robin Hwang's repository page for an example parameter scan on FACET-II. Click on the Fork button in the top-right corner of the page. This will create a copy of the repository under your own GitHub account.
After forking the repository, clone it to your local machine. Replace YOUR_USERNAME with your GitHub username.
git clone https://github.com/YOUR_USERNAME/facetii-parameter-scan.git
Change into the directory of the cloned repository.
cd facetii-parameter-scan
Our parameter scans require submodules that are essential for the project to work. Before creating an environment for this repository, fetch and initialize these submodules.
git submodule update --init --recursive
Compress your repository into a ZIP archive and name it facetii-parameter-scan.zip. You will use this archive later.
SSH into Perlmutter
The Perlmutter GPU partition includes nearly 1800 GPU nodes, each with 4 NVIDIA A100 GPUs, and the CPU partition includes over 3000 CPU nodes, each with 2 AMD Milan CPUs. The majority of Perlmutter's computational capability is in the GPU partition. Python users who wish to take advantage of this will need to adjust their code to run on GPUs.
Access the Perlmutter partition via ssh. Replace YOUR_USERNAME with your NERSC username.
ssh YOUR_USERNAME@perlmutter.nersc.gov
The terminal will prompt you for a password and one-time password. Type in your password to log in to Iris and (without any additional characters or whitespace) your one-time password.
Load in the nersc-python module to enable Python keyword recognition.
module load python
Creating an Environment
After accessing Perlmutter via SSH, create an environment for this code using mamba. Mamba is an alternative to conda that is faster and better at solving dependencies, but it does come with some pitfalls.
mamba env create -f environment.yml
PITFALL Ensure that both mamba and miniforge are installed on your system. In addition, there are a few dependencies that use pip to install, so make sure pip is installed.
PITFALL Mamba is only compatible with Windows, OS X, and Linux (ARM64 PPC64LE only). If you do not have one of these systems, simply use conda instead.
Activate your environment.
mamba activate Multifidelity
Run the following command to install the environment as a Jupyter kernel.
python -m ipykernel install --user --name env --display-name MyEnvironment
Install ipywidgets in your environment.
conda install -c conda-forge ipywidgets
Initiating a Session
Navigate to jupyter.nersc.gov and initiate a session. Select MyEnvironment from the list of available kernels to start using your new environment.
Upload facetii-parameter-scans.zip file from before into JupyterLab on NERSC.
Using the launcher, open a new terminal and unzip your archive.
! unzip ~/facetii-parameter-scans.zip -d ~/
Conclusion
This should be a quick guide for setting up an environment to run FACET-II parameter scans and using it in Jupyter on NERSC. You should now be able to run Training_Data_Example.ipynb and conduct a sample parameter scan up to PR10571, a screen in the injector. For more details on managing environments as Jupyter kernels at NERSC, refer to the official NERSC documentation. If you have any questions, feel free to contact Robin Hwang (robinny@slac.stanford.edu).