Page History
...
- includes the nvidia cudnn for deep learning frameworks like tensorflow
- adds paths to PATH, LD_LIBRARY_PATH, and CPATH so that you can work with the CUDA installation, and the nvidia cudnn
- for packages like tensorflow, that are compiled differently to work with the GPU, includes the GPU version of that package rather then the GPU CPU version
- presently, tensorflow is the only such package that is compiled differently for the GPU - that is all other packages in this environment are the same as the standard psana environment.
- packages like theano can be dynamically configured to use the GPU, so it is the same package between this gpu and non gpu environment
Using the cuDNN
Before using the cuDNN nvidia cudnn (by working with tensorflow or keras in the gpu environment, or configuring theano to to use it), register with the NVIDIA Accelerated Computing Development program at this link:
...
Presently, the GPU's are only available through interactive nodes. There is no batch management of them to assign GPU resources to users. Be mindful that other users on a node like psanagpu102 may be using the GPU.
The main issue is that GPU memory can become a scarce resource.
...
to see what other processes are on the gpu and how much memory they are using. Use
top
to identify the names of other users and communicate with them, or us, to manage multi-use issues.
Limit GPU Card Use
You If you are on a node with more than one GPU card, you can use cuda environment variables to restrict any CUDA based program, to only see a few of the GPU cards. If you are on a node that has two or more GPU cards (psanagpu101 is the only such node at this point) For example, if there are two cards, they will be numbered 0 and 1 by CUDA. You could do
export CUDA_VISIBLE_DEVICES=1
...
will start an interactive ipython session where tensorflow will only see device 1, and it . Tensorflow will call the one device it sees device 0.
Tensorflow: Limit GPU Memory Memory on a Card
With tensorflow, you can write your code to only grab the GPU memory that you need:
with tf.device('/gpu:0'): # this with statement may not be necessary config = tf.ConfigProto() config.gpu_options.allow_growth = True config.allow_soft_placement=True with tf.Session(config=config) as sess: # now your program, all variables will default to going on the GPU, and # any that shouldn't go on a GPU will be put on the CPU. |
Configuration Subject to Change
...
Overview
Content Tools