To-Do List

5/23/2012

• understand x/c kernel bottleneck with nvvp (lin)
• trying cufft to see what we gain (lin)
• use VO as convergence test case (aj)
• look at special-metric-weight convergence (aj)
• think about FFT cutoff (aj)
• GEAM, ZHERK (jun)
• build on hopper and xsede machines (jun/cpo)
  • generate benchmark strong-scaling plots for exx/rpa (no swapping!)
  • use std err to look for node-to-node "time variations"
• work on libxc (cpo)

5/9/2012

• rpbe kernel (lin)
  • does memcpyasync need cudamallochost?
  • fix stream behavior and try with 1,2,4,8,16 streams
  • understand stream behaviour with nvvp
• zher streams(jun)
  • in benchmark, have separately variable nstream/nw
  • can we see whether we have 4 or 16 streams?
  • understand stream behaviour with nvvp
• density mixing (aj)
• work on libxc (cpo)

5/2/2012

• looking at EXX bottleneck (rewriting) (jun)
• use cuda streams for small RPA systems (jun)
• libxc integration (cpo)
• understand MKL benchmark (jun/cpo)
• pycuda (cpo)
• understand RPBE kernel: (lin)
  • understand "double" problem
  • vary np, block_size, nstreams
  • loop testfunc many times
  • longer term: look at jussi/samuli kernel for ideas

4/25/2012

• looking at EXX bottleneck (rewriting) (jun)
• postpone work on ZHER stuff until we have news from INCITE (jun)
• talk to Frank about computing time applications (cpo)
• understand MKL benchmark (jun/cpo)
• libxc integration (cpo)

4/18/2012

• look at reduced-scope libxc example plus RPBE (lin)
• if there is time, benchmark the RPBE kernel (lin)
• zher performance improvement with multiple streams (jun)
• make INCITE version work (jun/cpo)
• move to libxc 1.2 (cpo)

4/11/2012

• libxc parallelization (lin)
• libxc integration (cpo)
• understand missing time in cublas mode (jun/cpo)
• how to put the gemm in PW mode in a fairly neat way (lin/cpo)
• start working on multiple-alpha kernel (MAZHER) (jun/cpo)
• work on INCITE proposal (jun/cpo)

3/28/2012

• gemm (lin)
• run pt3x3 (cpo)
• libxc (cpo, and lin if he finishes gemm)
• cher/fft (jun)
• fix gpu allocation (cpo)
• circular dependency problem with monkhorst_pack (cpo)
• mpi failure with cuzher (cpo)

3/21/2012

• batch queue for GPU machine (cpo)
• fft/gemm/gemv (lin/jun/cpo)
• single precision cher instead of zher? (jun/cpo)
• new libxc (cpo)
• fix libfftw detection (cpo)
• improve zher in cuda (long project, jun/cpo)
• move "expand" from python into C, post to mailing list? (lin)
• look at spin paired (cpo)
• run pt3x3 (cpo)

3/14/2012

• pycuda compatibility (cpo)
• private svn (cpo)
• try nvvp/transpose (or C60 with more grid points) for >5 minutes (lin)
• send mail to nvidia or list to understand why nvvp profile cuts off after 5 minutes (lin)
• understand bottleneck in get_wfs (jun)
• implement fft/gemv (cpo)
• is there a cuda library for trace like zgeev (cpo)
• run a 3x3x3 system to see if bottlenecks stay the same (cpo)
• driver hang status (cpo)
• understand how to fix gs.py bottlenecks in more detail (lin/cpo) using gpaw profiler:
  • pseudo density: density.py: self.calculate_pseudo_density(wfs) (cpo)
  • projections: overlap.py: wfs.pt.integrate(psit_nG, P_ani, kpt.q) (cpo)
  • RMM-DIIS: eigensolvers/rmm_diis.py: lots of lines (cpo)
  • projections: eigensolvers/rmm_diis.py: wfs.pt.integrate(dpsit_xG, P_axi, kpt.q) (lin)
  • calc_h_matrix: eigensolvers/eigensolver.py: H_nn = self.operator.calculate_matrix_elements, hamiltonian.xc.correct_hamiltonian_matrix (lin)
  • rotate_psi: eigensolvers/eigensolver.py (lin)

Accessing suncatgpu01 SVN

We have put a version of GPAW in a local SVN repository on suncatgpu01. To access it, use the following:

svn co svn://localhost svngpaw

You can put whatever you want for the last argument (local directory name).

General Topics

• Stanford CUDA course: http://code.google.com/p/stanford-cs193g-sp2010/
• (Everyone) Understand gpaw (read paper)
  • what other steps could we parallelize?
  • Can we do existing parallelization better? (e.g. use ideas in Todd's GPU papers)
• (Everyone) Go through CUDA tutorial here.
  • Understand blocks/threads/warps and how they map onto GPU hardware (details of which can be seen with "deviceQuery" command)
• (Lin) Find tool to measure:
  • memory bandwidth usage
  • gpu flops usage
• (Jun) :
  • Parallelize LCAO/planewave/RPA (zher performance?)? non-rmm-diis eigensolver?
  • merge with trunk?
• (cpo) :
  • Understand code flow
  • Understand where the ~23 cuda kernels are used
  • Understand which bottlenecks we need to tackle
• Do another gpu-gpaw install (to learn)
• Understand Pt 3x4x3 CPU/GPU difference versus 3x2x3 (performance scaling with system size)
• Can multiple CPU processes win by using the same GPU?
• Understand pycuda
• Understand gpaw interface to cuda (c/cuda subdirectory)
• Read CUDA programming manual here.
• Do all gpaw self-tests pass with GPUs?
• Can we get bigger bang-per-buck with GeForce instead of Tesla? (don’t need GPUDirect, maybe live with less memory/bandwidth? double precision worse)
• Understand cuda better:
  • Does Samuli use pinned memory correctly?
  • run/understand cuda a bandwidth benchmark
  • Could we use GPUdirect for MPI data transfer?
• Does the GPU performance scale with the product of gridpoints*bands? Might be a combinatorial effect with the bands, linear with the grid points?
• Duplicate Samuli results
• Update to most recent version in svn
• Understand where gpaw scaling maxes out for Pt 3x4x3
• Why is CO on 2 GPUs slower than on 8 CPUs?
• Can we do something less precise in the vacuum area? (fewer grid points?)
• Do we need a fatter interconnect for GPUs?

What We Learned at GTC

what we learned at gpu conference:

• how large blocks fit onto the SM (esp. wrt shared memory)
• nersc gpu cluster (dirac) usable?
• email addresses of fft/zher guy
• occupany spreadsheet
  • registers per thread
  • threads per block
  • shared memory per block
• access:
  • shared memory 10 clock ticks
  • global memory: 400*800 clock ticks
• l1*cache/shared memory size cnan be traded off (16/48kB)
• number of streams: 16 for 2.0 devices
• zher improves 50% in cuda 5 ("enable atomics")
• can try GEAM of ZHERK instead of ZHER
• blas2 functions memory bound, blas3 compute bound
• can maybe get access to zher source if we really need it
• 3d ffts might be good
• ucla gpu cluster available?
• openACC ony available for certain compilers (pgi, cray)
• new kepler/cuda5 features:
  • dynamic parallelism
  • hyper*Q for streams
  • slower clock speed more cores
  • mem 5x faster (maybe because of more reg space)
• startup time with multi*gpus is a known problem
• nvvp is dying: eclipse
  • much better information with nsight
• VASP person says we will win with 100x100x100 grid
• use single precision for pre-conditioning
• read the "best-practices" manual to understand nvvp profiling
• universal address space for transfering data
  • P2P for multi*gpus within a process
  • IPC for multi*gpus between processes