Subject: Re: CCL:Scaling of G98 on Clusters

From: "Andy Tucker" <[email-masked]>
Date: Fri, 31 Mar 2000 10:54:57 EST

Warning: Onageristic Estimate follows...
> this question goes to those who use Gaussian 98 (or maybe also other
> versions) on workstations clusters. 
> 

> Now, I ran the jobs with a-pinene and abietic acid (a mono- and a
> diterpene) with 1, 4, 8, 12 and 16 nodes to see how the setup scales.
> What I see is an almost linear scaling up to 4 nodes. With eight nodes
> the effective speedup is still nice with 6.41 where perfect behaviour
> obviously would be 8.00. But then it gets worse as you see and when
> going from 12 to 16 nodes the calculation even slows down. Here are the
> numbers. 
>
> Is this bad scaling to be expected? The graph on the gaussian web site
> shows the speedup of the calculations up to six nodes and I begin to
> understand why that is :-)

I'm just getting started with some parallel molecular dynamics 
calculations, so I can't claim to be an expert.  However, upon 
reading about the Gaussian scaling problem, it seems that this 
could be a general problem in computational chemistry 
calculations.

The program "GROMACS"  does molecular dynamics on large 
systems, and scales pretty well up to 8 nodes, and performance 
starts to slack off after that point. (see 
http://rugmd0.chem.rug.nl/~gmx/gmxbench.html )

My guess is that there is a similar cause for the problems observed 
in Gaussian and GROMACS.  GROMACS splits molecular 
systems into several pieces(slabs), one for each node.  Each node 
does a calculation on its piece of the molecule, then passes on 
information to neigboring pieces of the molecule for subsequent 
calculations.  As you increase the number of nodes, the size of the 
system treated by each node becomes smaller.  Depending on the 
size of the whole system, there will naturally be a point at which 
the calculations on each node take a short period of time relative to 
the time required for communication between nodes.  At that point 
the cluster begins to need more time for IO rather than 
calculations, and the scaling drops.  I would expect larger systems 
to scale better than smaller ones, and one can imagine having a 
system so small that it's not even worthwhile to use 2 processors.

I don't have any experience with ab initio calculations on clusters, 
but I would guess that the parallel implementation would be similar, 
splitting up the system into smaller pieces, with frequent 
correlation between nodes, so one would expect the same behavior.

Again, I'm making this up as I go along, but it seems reasonable, 
and I'd love to hear from someone who has more experience.
andy

Dr. W. Andrew Tucker            Queens College Chem. Dept.
[email-masked]              1900 Selwyn Ave.
(704)337-2317                   Charlotte, NC 28277

---

From: Rick Venable <[email-masked]>
Date: Sun, 2 Apr 2000 15:16:50 -0400 (EDT)

I have observed similar behavior for CHARMM recently-- reasonable
scaling for up to 8 nodes, but problems with e.g. 16 nodes.  This is
using MPI on either 100baseT or Gbit ethernet as the host-to-host
transport within a cluster of Intel-based machines running Linux (RH). 
It seems to be typical of tightly coupled parallel calculations, where
there is significant data passing between processing streams.  Ethernet
based Linux clusters seem to be I/O bound for more than 8 processors.

I suspect packet collisions may be an issue-- IP ethernet protocols tend
to degrade under heavy load.  One factor in this is that when 2 packets
arrive at a host interface at the same time (collide), BOTH get
rejected, and both originating hosts must resend the packet.  I'm a
little surprised that a high speed token ring interface hasn't been
developed and marketed for such dedicated, sustained high bandwidth
applications.  Myrinet does offer an alternative to TCP/IP ethernet,
which shows better scaling for 8 processors (I hope to test 16 sometime
soon), but the board cost is ca. $1000 US and requires custom switches
as well. 

Problem partioning (load balancing) and synchronization frequency are
other factors in this situation; all processors are limited to the rate
of the slowest process stream.  For parallel MD, where synchronization
is required on every time integration step (global sums, etc.), load
imbalances can have a significant impact.  A substantial effort was
expended into determining the best heuristic to use for partitoning the
nonbond pair list (list sequence, spatial decomposition, etc.) within
CHARMM to achieve good load balancing.  The frequent synchronization
means, however, that all nodes are trying to communicate and obtain
results simultaneously on each timestep, a scenario which invites
collisions on ethernet transports. 

--
Rick Venable                  =====\     |=|    "Eschew Obfuscation"
FDA/CBER Biophysics Lab       |____/     |=|
Bethesda, MD  U.S.A.          |   \    / |=|  ( Not an official statement or
[email-masked]          |    \  /  |=|    position of the FDA; for that,
[email-masked]              \/   |=|    see   http://www.fda.gov  )