View Source

GPAW Convergence Behavior

A talk given by Ansgar Schaefer studying convergence behaviour for rutiles is here (pdf).

General suggestions for helping GPAW convergence are here.

A discussion and suggestions for converging some simple systems can be found here.

Other convergence experience:

System	Who	Action
Graphene with vacancy	Felix/JensH	Increase Fermi Temp from 0.1 to 0.2, use cg
Graphene with vacancy	cpo	change nbands from -10 to -20, MixerDif(beta=0.03, nmaxold=5, weight=50.0)
Enzyme-inspired CO2 reduction	grabow	use Davidson solver (faster as well?)

GPAW Memory Estimation

The get a guess for the right number of nodes to run on for GPAW, run the
following line interactively:

gpaw-python <yourjob>.py --dry-run=<numberofnodes>
(e.g. gpaw-python graphene.py --dry-run=16)

Number of nodes should be a multiple of 8. This will run quickly
(because it doesn't do the calculation). Then check that the
following number is <3GiB:

Memory estimate
---------------
Calculator  574.32 MiB

Building a Private Version of GPAW

Use svn to check out the version of GPAW that you want to use
copy /afs/slac/g/suncat/bin/privgpaw.csh into whatever directory you like
edit the two variables GPAW_BASE ("base" GPAW release that you want to re-use for numpy, mpi etc.) and GPAW_HOME (directory where you checked out GPAW)

Use the commands:

privgpaw.csh build
privgpaw.csh test
privgpaw.csh bsub <arguments>

Some notes:

The syntax for the "bsub" option is identical to the gpaw-bsub command.
The above assumes that the base release is compatible with your checked out GPAW version. Talk to cpo if you have questions about this.
The above doesn't include a private version of ASE. When that becomes important we will add it.

{htmlcomment}

These directions are adapted from the GPAW website, and you should check that to make sure this still makes sense. These particular instructions are in order to have it integrate well with the suncat farm systems.

The user should already have an svn version of ASE somewhere in their home directory -- the bleeding edge version of GPAW will often only work with the most up-to-date version of ASE.

Create a space where custom checked-out versions of software live. Probably:

{code}
mkdir /nfs/slac/g/suncatfs/${username}/usr/
{code}

(where $\{username\} is your username). From now on this will be referred to as $\{usr\}. You can optionally create an environment variable in your .cshrc to make the rest of the steps cut-and-paste. Add to your .cshrc:

{code}
setenv usr /nfs/slac/g/suncatfs/\{username}/usr/
{code}

with your username in place of $\{username\}. (Then log out and back in.) Create a directory to put a checked-out version of gpaw and set this as the variable $\{GPAW_HOME\}

{code}
mkdir -p \${usr}/lib/svn/gpaw
setenv GPAW_HOME \${usr}/lib/svn/gpaw
{code}

Check out the latest version of GPAW.

{code}
svn checkout https://svn.fysik.dtu.dk/projects/gpaw/trunk ${GPAW_HOME}
{code}

Build the C extensions.

{code}
cd ${GPAW_HOME}
python setup.py build_ext |& tee build_ext.log
{code}

(Consult [https://wiki.fysik.dtu.dk/gpaw/devel/developer_installation.html#developer-installation] to make sure that the proper files have been built.)

Install the latest GPAW setups. Create and enter a new directory.

{code}
mkdir -p ${usr}/lib/gpaw-setups
cd ${usr}/lib/gpaw-setups
{code}

Visit the GPAW setup webpage and get the version number of the latest version of the setups. As of this writing, it is 0.8.7929. Modify the line below with the correct version number, then download and unpack them with

{code}
wget http://wiki.fysik.dtu.dk/gpaw-files/gpaw-setups-<version>.tar.gz
tar zxf gpaw-setups-<version>.tar.gz
rm gpaw-setups-<version>.tar.gz
{code}

Create a path for the environment settings (if it doesn't exist).

{code}
mkdir ${usr}/env
{code}

If you don't already have one, create an environment file to your SVN version of ASE. Create a file called $\{usr\}/env/ase-svn which contains the two lines:

{code}
#!/bin/csh
setenv PYTHONPATH /path/to/ase/svn:${PYTHONPATH}
{code}

where /path/to/ase/svn is the path to your SVN version of ASE (e.g., $\{usr\}/lib/svn/ase). Next create an environment file to your SVN version of GPAW, called $\{usr\}/env/gpaw-svn, which contains the following lines:

{code}
#!/bin/csh

setenv GPAW_HOME ${usr}/lib/svn/gpaw
setenv GPAW_PLATFORM `python -c "from distutils import util, sysconfig; print util.get_platform()+'-'+sysconfig.get_python_version()"`
setenv PYTHONPATH ${GPAW_HOME}:${PYTHONPATH}
setenv PYTHONPATH ${GPAW_HOME}/build/lib.${GPAW_PLATFORM}:${PYTHONPATH}
setenv PATH ${GPAW_HOME}/build/bin.${GPAW_PLATFORM}:${GPAW_HOME}/tools:${PATH}
setenv GPAW_SETUP_PATH ${usr}/gpaw-setups/gpaw-setups-XXXX:${GPAW_SETUP_PATH}
source ${usr}/env/ase-svn
{code}

Note in the above that you will need to substitute in (for XXXX) the correct directory for the version of GPAW setups you have installed.

Lastly you will need to make an executable submission script. If you don't have one already, create a directory that contains your executables:

{code}
mkdir ${usr}/bin
{code}

To make sure you can execute files from this directory, you will need to put the path to this in your PATH environment. Add the following line to your .cshrc file:

{code}
setenv PATH ${usr}/bin:${PATH}
{code}
Now create your submission script that will be used to submit gpaw jobs from your svn version. Create a file called $\{usr\}/bin/gpaw-svn-bsub and place the following three lines in it.

{code}
#!/bin/tcsh
source ${usr}/env/gpaw-svn
/afs/slac/g/suncat/bin/dobsub -a openmpi -R "span[ptile=8]" gpaw-python $argv
{code}

Then change the mode of this file to be executable.

{code}
chmod u+x ${usr}/bin/gpaw-svn-bsub
{code}

Run the tests as described on the GPAW website:
{code}
https://wiki.fysik.dtu.dk/gpaw/install/installationguide.html#running-tests
{code}

If everything is working right, you should be able to submit scripts using your svn version with

{code}
gpaw-svn-bsub <scriptname>.py
{code}

{htmlcomment}