Page History

...

MPICH on Ethernet: no noticeable peaks

To run the test:

OpenMPI with Infiniband: 

Clone psana environment then remove mpi4py, mpich, and mpi.

Build openmpi on drp nodes (drp-tst-dev011 was used for this test). No special flag needed just use --prefix to put the build somewhere.

Do conda build for mpi4py (see recipe on relmanage/recipe) by pointing build.sh script to the openmpi build.

Install this new mpi4py to the cloned conda env.

Running it with (for example), 

~/tmp/4.0.0-rhel7/bin/mpirun --hostfile openmpi_hosts --mca btl_openib_allow_ib 1 run_slac.sh

where ~/tmp/4.0.0-rhel7/ is the --prefix used to build openmpi.

(ps-1.2.2-openmpi) monarin@drp-tst-dev011 (master *) psana2 $ cat openmpi_hosts drp-tst-dev011 slots=1

drp-tst-dev012 slots=12

drp-tst-dev013 slots=20

drp-tst-dev014 slots=20

drp-tst-dev015 slots=20

drp-tst-dev016 slots=20

drp-tst-dev017 slots=20

(ps-1.2.2-openmpi) monarin@drp-tst-dev011 (master *) psana2 $ cat run_slac.sh 

#!/bin/bash

export PS_SMD_NODES=12

source $HOME/lcls2/setup_env.sh

conda activate ps-1.2.2-openmpi

python dev_bd.py

MPICH on ethernet

(ps-2.1.2) monarin@drp-tst-dev011 (master *) psana2 $ /reg/g/psdm/sw/conda2/inst/envs/ps-2.1.2/bin/mpirun -f mpich_hosts ./run_slac.sh

(ps-2.1.2) monarin@drp-tst-dev011 (master *) psana2 $ cat mpich_hosts

drp-tst-dev011:1

drp-tst-dev012:12

drp-tst-dev013:20

drp-tst-dev014:20

drp-tst-dev015:20

drp-tst-dev016:20

drp-tst-dev017:20

(ps-1.2.2-openmpi) monarin@drp-tst-dev011 (master *) psana2 $ cat run_slac.sh 

#!/bin/bash

export PS_SMD_NODES=12

source $HOME/lcls2/setup_env.sh

python dev_bd.py