Subject: SUMMARY: Linux parallel scaling

From: Rick Venable <[email-masked]>
Date: Fri, 21 Dec 2001 16:04:26 -0500

We have Myrinet on some nodes, and it scales very well for MD codes like
CHARMM and AMBER.  However, there are far more ethernet nodes available,
and there never seems to be enough computer time available ...

I should also note that for CHARMM, performance varies greatly with the
compiler used-- we have observed roughly 25-45% speedup for executables
produced with the PGI compiler wrt. g77.  The largest speedups were
observed for simulations with Particle-Mesh Ewald, probably because g77
doesn't optimize the 3D FFT very well (ca. 18 min on one proc for the
test below).

Prior to the summary, some background-- with dual P3 or AMD nodes, we
observed the following for a ca. 19K atom MD test with PM Ewald using a
CHARMM executable compiled with pgf77 and MPICH (completion time for 100
steps, in minutes; N processors):

  P3-866 2.2  P3-866 2.4  AMD p1400 2.4
  ----------- ----------- -----------  
N  CPU   WALL  CPU   WALL  CPU   WALL
1 10.07 10.07 10.36 10.37  5.67  5.68
2  6.74  7.37  7.25  7.88  3.55  4.13 <--
4  4.11  5.12  4.87  6.52  3.24  5.35 <--
         ****        ****  

Note the WALL clock time in particular; there is a continuing decrease
thru 4 processors for the P3-866 nodes, but it is much greater for the
Linux 2.2.x-tcpfix kernel.  Note that 2 procs is better than 4 for the
AMD p1400 nodes-- but this may be due in part to rough doubling of
processor speed, such that the MD simulations are essentially I/O bound.

Excerpts from the replies I received are below, following my signature.

=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=
Rick Venable
FDA/CBER/OVRR Biophysics Lab
1401 Rockville Pike    HFM-419
Rockville, MD  20852-1448  U.S.A.
(301) 496-1905   [email-masked]
ALT email:  [email-masked]
=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=

	Bogdan Costescu <[email-masked]>

I've just updated the software on one of our clusters, that we use
mainly for running CHARMM, to Red Hat Linux 7.2 (but using kernel 2.4.16
> from RawHide) and tested latest versions of LAM-MPI and MPICH. As I
expected (got the same behaviour at every test - about every 6 months
for the last 2 years), LAM was significantly better - about 25% faster.
Also I couldn't notice any significant speed change between this test
and the previous ones (which used Red Hat Linux 6.x with official Red
Hat kernels up to 2.2.19 and then 2.2.20; I only applied the TCP patch
to 2.2.16, with >= 2.2.19 kernels I couldn't see a significant
difference).

I could also get something more significant by compiling CHARMM without
GENCOMM flag; this way it uses collective operations from the MPI
library;  the downside is that it's only allowed to run on power of 2
number of nodes (I hope that I got this right, as I said, it was a year
ago...)

	David Konerding <[email-masked]>

Are you using TCP/IP over 100BaseT ethernet?  If so, then I don't really
expect you will get very good scaling.  My experience with AMBER and
CHARMM is that they get about 2X for 4 CPUs at best, and that the
bottleneck is within the MPI implementation and the performance of
ethernet.  The latencies are just too high.  CHARMM and AMBER both scale
much better if you get a good interconnect like Myrinet and use their
MPI libraries.

	Ivan Rossi <[email-masked]>

I had a very similar problem using Scyld Beowulf, that uses a custom
version of MPICH, with GROMACS. The speedup was going up as sqrt(nodes).
The problem has been solved by using LAM-MPI (www.lam-mpi.org) and
replacing Scyld with a standard redhat distibution. It seems that it is
a better implementation of MPI.

however scaling is not very good for parallel MD on fast Ethernet, data
packets are small, and frequently exchanged. On the contrary I have seen
linear scaling up to 16 nodes for MD codes on Linux cluster equipped
with low-latency connections such as Myrinet or scali dolphin, but it is
much more expensive.