Versions Compared

Old Version 18

changes.mady.by.user Unknown User (davidsch)

Saved on

compared with

New Version 19

changes.mady.by.user Unknown User (davidsch)

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

 

Here is an example

In [1]: import anarelinfo
In [2]: anarelinfo.version
Out[2]: 'psana-conda-1.0.3'
In [3]: anarelinfo.pkgtags
Out[3]:
{'AppUtils': 'V00-07-00',
 'CSPadPixCoords': 'V00-03-30',
...

 

GPU Work

LCLS has some GPU resources with some software setup for use. See table below

nodeCUDAGPU card(s)RAMCompute
Capability 		notes
psanagpu101     
psanagpu1027.5Tesla K4012 GB3.5

This is the only card we have with a modern enough compute capability for deep learning frameworks that rely on the nvidia cudnn (like tensorflow)

psanagpu103     
      

We are still developing infrastructure and configuration for these nodes, but presently, if one does

ssh psanagpu102
source conda_setup --dev --gpu

then you will be activating a python 2.7 conda environment for working with the GPU. It is mostly the same as the main environment with psana, but has these differences:

  • includes the nvidia cudnn for deep learning frameworks like tensorflow
  • adds paths to PATH, LD_LIBRARY_PATH, and CPATH so that you can work with the CUDA installation, and the nvidia cudnn
  • for packages like tensorflow, that are compiled differently to work with the GPU, includes the GPU version of that package rather then the GPU version
    • presently, tensorflow is the only such package that is compiled differently for the GPU - that is all other packages in this environment are the same as the standard psana environment.
    • packages like theano can be dynamically configured to use the GPU, so it is the same package between this gpu and non gpu environment 

Using the cuDNN

Before using the cuDNN (by working with tensorflow in the gpu environment, or configuring theano to use it), register with the NVIDIA Accelerated Computing Development program at this link:

 https://developer.nvidia.com/accelerated-computing-developer

Per the nvidia cuDNN license, we believe all users must register before using it, but don't worry, the nvidia emails (if you opt to receive them) are quite interesting! (smile) 

Shared Resource

Presently, the GPU's are only available through interactive nodes. There is no batch management of them to assign GPU resources to users. Be mindful that other users on a node like psanagpu102 may be using the GPU. The main issue is that GPU memory can become a scarce resource.

Make use of the command

nvidia-smi

to see what other processes are on the gpu. Use 

top

to identify the names of other users and communicate with them, or us, to manage multi-use issues.

Limit GPU Card Use

You can use cuda environment variables to restrict any CUDA based program, to only see a few of the GPU cards. If you are on a node that has two or more GPU cards (psanagpu101 is the only such node at this point) You could do 

export CUDA_VISIBLE_DEVICES=1

and any command you run will only see that one GPU. Likewise, to just start one process with a limited view, do

CUDA_VISIBLE_DEVICES=1 ipython

will start an interactive ipython session where tensorflow will only see device 1, and it will call it device 0.

 

Tensorflow: Limit GPU Memory Memory on a Card

With tensorflow, you can write your code to only grab the GPU memory that you need:

with tf.device('/gpu:0'):
    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    config.allow_soft_placement=True
    with tf.Session(config=config) as sess:
         # now your program, all variables will default to going on the GPU, and
         # any that shouldn't go on a GPU will be put on the CPU.

 

Configuration Subject to Change

At this point there are very few people using the GPU and the configuration of GPU support is subject to change. Presently the gpu conda environment is only built in the development rhel7 conda installation (thus the --dev switch for conda_setup above).