Most Recent GPAW Versions

Farm

New Setups (0.9.9672)

Old Setups (0.8.7929)

suncat/suncat2

gpawv40

gpawv41

suncat3

gpawv40a

gpawv41a

Submitting Batch Jobs

gpaw-bsub myscript.py ("old/stable" version)
gpaw-ver-bsub <version> myscript.py (user-selected version)

GPAW Convergence Behavior

5/1/2013: Jun Yan reports better convergence behavior with a new version of gpaw (v40/v41) that has an improved eigensolver. Use code like this:

from gpaw.eigensolvers.rmm_diis_new import RMM_DIIS_new   
calc = GPAW(?...,eigensolver=RMM_DIIS_new(niter=6),...)                                                                                   

A talk given by Ansgar Schaefer studying convergence behaviour for rutiles is here (pdf).

General suggestions for helping GPAW convergence are here.

A discussion and suggestions for converging some simple systems can be found here.

Other convergence experience:

System

Who

Action

Graphene with vacancy

Felix Studt

Increase Fermi Temp from 0.1 to 0.2, use cg

Graphene with vacancy

Chris O'Grady

change nbands from -10 to -20, MixerDif(beta=0.03, nmaxold=5, weight=50.0)

Nitrogenase FeVCo for CO2 reduction

Lars Grabow

use Davidson solver (faster as well?), although later jvarley said MixerSum

Several surfaces

Andy Peterson

Broyden mixer with Beta=0.5

TiO2

Monica Garcia-Mota

MixerSum(0.05,6, 50.)

MnxOy

Monica Garcia-Mota

Broyden MixerSum

Co3O4

Monica Garcia-Mota

Davidson eigensolver, MixerSum(beta=0.05, nmaxold=5, weight=50) or MixerSum(beta=0.05, nmaxold=6, weight=100)

MnO2 with DFT+U U=+2eV

Monica Garcia-Mota

Marcin suggests we disable the DipoleCorrectionPoissonSolver (not yet tested)

MnO2 with DFT+U U=+2eV

Monica Garcia-Mota

Henrik Kristofferson suggests: convergence is easier with high U (U=4eV) and then
one can shift to preferred value

MnO2 with DFT+U U=+2eV

Monica Garcia-Mota

(from Heine) increase U in steps of say 0.1 (or smaller) and reuse the density and/or wave functions from the previous calculation? This tends to reduce the problem of being trapped in meta-stable electronic states, and it also makes convergence easier. Monica later reported that this helped.

Cu

Ask Hjorth Larsen

first mixer parameter should probably be 0.1 for faster convergence, because it has a low DOS at the Fermi level. (Other transition metals may require lower values.)

N on Co/Ni (with BEEF)

Tuhin

rmm-diis and MixerSum(beta=0.1, nmaxold=5, weight=50)

Other Tricks:
  • To speed up the poisson solver, use something like gpts=h2gpts(h=0.18, atoms.get_cell(), idiv=8) to get a nicely divisible-by-large-power-of-2 grid. This helps the "multi grid" poisson solver.
  • experiment with Fermi smearing to help improve convergence
  • experiment with number of empty bands to help improve convergence
  • be sure to specify nbands, otherwise GPAW will add "plenty" of bands which is very expensive in FD calculations. nbands=6*[number of atoms] should be more than enough.

GPAW Planewave Mode

Jens Jurgen has a post here that discusses how to select plane wave mode in your script.

It looks like we have to manually turn off the real-space parallelization with the keyword:

parallel={'domain': 1}

In planewave mode I believe we also can only parallelize over reduced k-points, spins, and bands. We have to manually set the right numbers for these to match the numbers of CPUs.

  • To get parallelization over bands, we can at the moment only use the rmm-diis eigensolver (cg and davidson don't work).
  • The number of bands must be divisible by the number of CPUs, according to Jens Jurgen.
  • At the moment there is no dipole correction in planewave mode.
  • Density mixing is still done in real-space

GPAW Geometry Optimizations

Thoughts from Lin and Thomas: With GPAW one can do geometry optimizations a factor of 10 faster in LCAO mode (with smaller memory requirements). Then it's necessary to "tweak" the optimization with a little bit of running in FD mode.

Plus, LCAO mode has the added feature that convergence is typically easier, according to Heine.

I think it's difficult to automate the above process in one script, since the number of cores required for LCAO is typically lower than FD (because of the lower memory usage).

But if you're limited by CPU time when doing GPAW optimizations it might be worth keeping the above in mind.

AJ adds: I would also warn against using LCAO as an initial guess for NEB calculations. I have tried this for 2 different systems and it turned out to be a tremendous waste of time. The NEB did not converge much faster with LCAO, and when I used the LCAO images as an initial guess for finite difference mode it still took several restarts to converge. I have had better luck using decreased k-point/grid spacing and relaxed convergence criteria as an initial optimization for NEBs.

GPAW Memory Estimation

The get a guess for the right number of cores to run on for GPAW, run the following line interactively:

gpaw-python <yourjob>.py --dry-run=<numberofcores>
(e.g. gpaw-python graphene.py --dry-run=16)

Number of cores should be a multiple of the number-of-cores-per-node here. The above will run quickly (because it doesn't do the calculation). Then check that the following number fits within the memory-per-core listed here :

Memory estimate
---------------
Calculator  574.32 MiB

Tips for Running with BEEF

If you use the BEEF functional:

  • use xc='BEEF-vdW'
  • the code parallelizes over 20 cores (because of the way the VDW contribution is calculated)
  • you typically need a lot of memory, even for small calculations. For larger calculations you sometimes have to use more than 20 cores, just to get enough memory (you don't get additional parallelization).
  • the GPAW memory estimates are incorrect
  • it is typically best to run on the suncat2 (more memory per core) or, even better, the suncat3 farm (more memory per core, plus a much faster infiniband node-node interconnect)
  • use the suncat3 farm, which has a fast (4GB/s) infiniband interconnect.

Building a Private Version of GPAW

  • Use svn to check out the version of GPAW that you want to use (described here).
  • copy /afs/slac/g/suncat/share/scripts/privgpaw.csh into whatever directory you like
  • edit the two variables GPAW_BASE ("base" GPAW release that you want to re-use for numpy, mpi etc.) and GPAW_HOME (directory where you checked out GPAW)
  • Use the commands:
    ./privgpaw.csh build (build)
    ./privgpaw.csh build-rh6 (build for the suncat3 RedHat6 farm. can use same source directory as build for old farms)
    ./privgpaw.csh test (run gpaw self-tests)
    ./privgpaw.csh gpaw-bsub <arguments> (submit batch job)
    ./privgpaw.csh gpaw-bsub-rh6 <arguments> (submit batch job to suncat3 RedHat6 farm)
    ./privgpaw.csh <cmd> (run <cmd> interactively, using privgpaw.csh environment.  e.g. "gpaw-python junk.py")
    

Some notes:

  • The syntax for the "bsub" option is identical to the gpaw-bsub command described here.
  • The above assumes that the base release is compatible with your checked out GPAW version. Talk to cpo if you have questions about this.
  • The above doesn't include support for a private version of ASE. When that becomes important we will add it.

Versions

Version

Date

Comment

gpawv1

9/14/10

gpaw 0.7.6383, first attempt at SUNCAT gpaw installation

gpawv2

9/21/10

move from standard lapack libraries to acml,blacs,scalapack to improve performance

gpawv3

9/30/10

move from ase 3.4.0 to 3.4.1

gpawv4

10/15/10

check out a gpaw svn branch for hannesj/aleksandra

gpawv5

10/15/10

copy of v3, add scipy

gpawv6

10/22/10

copy of v5, move GPAW to head for venkat (gpaw 0.8.0)

gpawv7

11/9/10

gpaw 0.7.2.6974, switch from ifort v9 to gcc 4.1.2, simplify setupenv

gpawv8

11/16/10

use goto2 blas and open64 to improve performance

gpawv9

12/9/10

copy of gpawv8, but HEAD of gpaw/ase to fix non-orthorhombic problem for strabo/pgmoses
also change gotoblas flags to try to improve performance

gpawv10

12/9/10

copy of gpawv9, but put in numpy site.cfg to try to use acml for numpy

gpawv11

1/10/11

copy of gpawv9, a special version for hildur to use

gpawv12

6/27/11

copy of gpawv9, a special version for jewe to use (preliminary BEEF version)

gpawv13

7/8/11

softlink to gpawv9, change setupenv to use torque/maui aware openmpi

gpawv14

8/10/11

copy of gpawv9, update to HEAD of gpaw svn, and HEAD of ase svn (for wulff construction) requested by Lin

gpawv15

10/5/11

test version built with icc/mkl

gpawv16

10/12/11

version like gpawv14, but with python2.7 so we can include hdf5 support for Lin. Only serial version functional with /opt/TWWfsw/python27/bin/python

gpawv17

11/16/11

copy gpaw/ase from v9, but use icc/ifort/mkl to improve performance

gpawv18

1/3/12

copy gpaw/ase from v17, but mkl 10.3 to fix LD_PRELOAD hack

gpawv19

1/20/12

like gpawv17, but GPAW from head of SVN to get first BEEF release, also most recent setups, and more recent ASE

gpawv20

1/21/12

mkl 10.3 with scalapack, gpaw from head of SVN, newer ASE, new setups. serial version broken because removed static linking?

gpawv21

2/1/12

uses v17 executables, but adds head of ASE and version 0.3.2 of CMR to PYTHONPATH (requested by AJ)

gpawv22

2/6/12

like v19 but uses GPAW/ASE from head of SVN to pickup BEEF error estimation. also enables scalapack, CMR

gpawv23

3/26/12

like v22 but uses GPAW v0.9, required ASE 3.6.0, required CMR 0.3.3.580

gpawv24

4/27/12

like v23 but uses head of GPAW and new ASE to pick up fix for vdW parallel NEB

 

6/22/12

NOTE THAT GPAW VERSIONS OLDER THAN 25 WILL NO LONGER WORK BECAUSE OF PYTHON-LIBS SECURITY PATCH PROBLEM

gpawv25

6/20/12

copy of v22 but rebuilt to fix python-libs patch problem

gpawv26

6/20/12

copy of v17 but rebuilt to fix python-libs patch problem

gpawv26a

11/16/12

(suncat3 farm only) copy of v26 ("old/stable") gpaw, but some tests fail still

gpawv27

6/20/12

copy of v24 but rebuilt to fix python-libs patch problem

gpawv27a

11/16/12

v27 rebuilt for suncat3 farm (softlink to v35)

gpawv28

6/22/12

copy of v24 but try python2.7 with updated numpy/scipy/matplotlib. gtk graphics broken

gpawv29

7/10/12

copy of v16 but rebuilt gpaw because of the python-libs problem. only serial version functional with /opt/TWWfsw/python27/bin/python

gpawv30

7/23/12

head of GPAW svn to get FixedOccupations class in occupations.py for Joel. But turned out v27 already had that. Also newer ASE. DISABLED this because wasn't optimally built.

gpawv31

10/12/12

same as V30, but with latest official setups. DISABLED this because need to update to latest GPAW for latest setups.

gpawv32

11/7/12

head of GPAW svn but with latest official setups

gpawv32a

11/14/12

v32 rebuilt for suncat3 farm (softlink to v33)

gpawv33

11/14/12

(suncat3 farm only) like v32, but rebuilt for Sandy Bridge and RHEL6

gpawv34

11/14/12

(suncat3 farm only) like v33, but uses older setups DISABLED because saw strange results

gpawv35

11/16/12

(suncat3 farm only) copy of v27 gpaw, so we can use older setups

gpawv36b

11/16/12

(gpu farm only) checkout of Jun's branch rpa-gpu-expt built for the new gpu farm. old setups.

gpawv37b

1/28/13

(gpu farm only) checkout of Samuli's cuda branch built for the new gpu farm. old setups.

gpawv38

2/1/13

build of libxc 2.0.0 branch for doylead/alevoj so they can use HSE.

gpawv39

2/25/13

copy of v27 with newer ASE 3.6.1.2981

gpawv40

3/24/13

latest gpaw from trunk to fix get_all_electron_density problem. new setups. two tests fail running with -j 8, but work running tests without -j 8

gpawv41

3/24/13

like v40, but old setups.

gpawv42

4/18/13

copy of v29 environment to pick up hdf5 for Lin. parallel version broken. latest gpaw from trunk to fix get_all_electron_density problem. old setups (so self-tests don't work). still need to use /opt/TWWfsw/python27/bin/python, I believe

gpawv43

6/28/13

first version built with new libxc

gpawv44

10/7/13

like v43, but with newer ASE for chank to pick up new minimahopping.py

  • No labels