Jupyter is a web based analysis and coding environment. It supports multiple different programming languages, but is mostly centered around python development. The main advantage over standard IDEs is that it provide immediate code execution and inline graphics - ie you can interactively explore data.
SLAC implements JupyterHub at http://jupyter.slac.stanford.edu. This provides a central point of access to your jupyter environment using SLAC credentials and access to data stored on SLAC GPFS.
What is somewhat unique to SLAC's implementation of jupyter is that we wish to:
- provide an environment for different experiments to utilise our infrastructure with minimal hassle.
- provide a means for users to run their own jupyter environments (their analysis environment, with their down dependencies)
The current implementation utilises kubernetes and docker images to provide the above functionality.
Link to tutorials
Jupyter on Batch
In order to provide a consistent environment across the web-based jupyter and batch systems, our currently recommendation is to develop your jupyter environment in Docker and then convert the images to Singularity. From there, we can integrate into our modulefiles system directly in order to provide command line access (and hence batch).
There is a specific module called
slac-ml that provides a prebaked Singularity image derived from the jupyterhub image.
When we load a module, it will override certain environments so that we can now use the 'application' defined in the modulefile:
In order to run on batch, we will use this modulefile system and submit jobs by creating a text file (eg
You can then submit the job to batch via
and you can monitor the job with
We recommend either using nbconvert or papermill to provide parameter access to notebooks.