Subject: RE: Scaling of G98 on Clusters

From: "Greg Lindahl" <[email-masked]>
Date: Fri, 31 Mar 2000 00:24:04 -0500

> Is this bad scaling to be expected?

Yes. Not only does g98 use Linda, which is hard to optimize, it's doing work
decomposition, so it ought to scale at best like Charmm does. Charmm goes
out to about 32 alpha nodes with myrinet before it begins to turn over, so
it shouldn't surprise you that ethernet turns over faster. I should
hopefully have some comparative benchmarks soon: alpha vs. intel, gigE vs.
Myrinet vs. fast ethernet.

-- g

---

From: "Phillip Matz" <[email-masked]>
Date: Thu, 30 Mar 2000 21:38:28 -0800

Hello!

Indeed I see very similar results in scaling G98 CIS(Direct) calculations.
My beowulf is similarly configured (450MHz nodes not 550).  Using the simple
equations regarding Amdahl's law and IPC provided in Robert G. Brown's web
article "So, You Want to Build a Beowulf" and varying a few parameters
(10Mbps vs 100Mbps, etc.) I was able to extract a few seemingly "important"
numbers concerning some of the G98 linda-executables.

I believe there is a good marketing reason why the white-paper regarding the
scalability of gaussian code on linda is reported for RHF(direct).  My
analysis on G98 RHF(direct) (the executable is 502.exel) indicates that the
serial code fraction (Ts in Amdahl's law) is around 1%, and for my
particular beowulf the IPC was approximately 2ms/node/second; with such a
low Ts one can't help but to expect some decent granularity and ok
scalability.  My field of research (I'm a grad student at Washington State
University) is spectroscopy so I'm particularly interested in Excited State
calculations, which G98 facilitates via CIS calculations.  My analysis on
G98 CIS(direct) (the executable is 914.exel) indicates that the serial code
fraction is around 8%!!  This of course leads to very poor G98 scaling when
I run a CIS calculation, with the peak scaleup being around 6 when 12 nodes
are being used and decreases thereafter just as you have noticed.

I don't use G98 for DFT calculations, but based on your observations I would
suspect that the G98 linda executables (.exel) being called from your route
file also have a high percentage of serial code in them.  I would also like
to point out that while the noted decrease in scalability can be alleviated
through channel bonding or faster networking, the maximum scaleup is
severely restricted due to the serial code fraction (Amdahl's law).

And one last comment, I was completely amazed at how easy and accurate it
was to model my G98 beowulf's performance based on the simple equations
found in Dr. Brown's beowulf profiling guide - so thank you Dr. Brown!

Respectfully,
Phillip Matz

----- Original Message -----
From: "Eugene Leitl" <[email-masked]>
To: <[email-masked]>
Sent: Thursday, March 30, 2000 6:25 PM
Subject: CCL:Scaling of G98 on Clusters

> From: Christoph Steinbeck <[email-masked]>
> Sender: "Computational Chemistry List" <[email-masked]>
> To: ccl <[email-masked]>
> Subject: CCL:Scaling of G98 on Clusters
> Date: Thu, 30 Mar 2000 19:09:48 +0200
>
> Greetings everybody,
>
> this question goes to those who use Gaussian 98 (or maybe also other
> versions) on workstations clusters.
>
> I'm currently in the process of evaluating g98 with linda my new 16-node
> linux cluster. Each node is a 550 Mhz Pentium III with 128 MB of RAM and
> a 6 GB hard disk. Communication is done via a 100 MBit ethernet and the
> 16 nodes are separated from the institutes network by a switch.
>
> The calculations I'm interested in are of the type indicated by the
> example listed below, which essentially are composite jobs with geometry
> optimizations on B3LYP/6-31G(d) level followed by a carbon shift
> calculation on the same level.
>
> Now, I ran the jobs with a-pinene and abietic acid (a mono- and a
> diterpene) with 1, 4, 8, 12 and 16 nodes to see how the setup scales.
> What I see is an almost linear scaling up to 4 nodes. With eight nodes
> the effective speedup is still nice with 6.41 where perfect behaviour
> obviously would be 8.00. But then it gets worse as you see and when
> going from 12 to 16 nodes the calculation even slows down. Here are the
> numbers.
>
> Nodes Effective speedup
> 1 1
> 4 3,93
> 8 6,41
> 10 8,55
> 12 8,67
> 16 7,9
>
> This is with a-pinen (C10H16).
> When more than doubling the size of the molecule with abietic acid
> (C20H30O2) the tendency is a little better (8 nodes take 41 h, 16 nodes
> take 30 h).
>
> Is this bad scaling to be expected? The graph on the gaussian web site
> shows the speedup of the calculations up to six nodes and I begin to
> understand why that is :-)
> In many publications on beowulf clusters you find the statement that in
> many cases one can expect near linear scaling of those cluster up to 16
> nodes. I understand that  this highly depends on the type of
> calculation, so am I stuck with my problem, or are there ways to
> optimize the behaviour by tuning the settings?
>
> BTW, I'm using the standard (install time) settings for g98 and linda.
>
> Any comments are very much appreciated.
>
> Cheers,
>
> Chris
>
> --
> Dr. Christoph Steinbeck (http://www.ice.mpg.de/departments/ChemInf)
> MPI of Chemical Ecology, Tatzendpromenade 1a, 07745 Jena, Germany
> Tel: +49(0)3641 643644 - Fax: +49(0)3641 643665
>
> What is man but that lofty spirit - that sense of enterprise.
> >... Kirk, "I, Mudd," stardate 4513.3..
>
> ----------------
> %Chk=abieticacid
> #T B3LYP/6-31G(d) Opt Test
>
>   Name: Z:\gaussian\abietinsaeure.mol
>   MOPAC file created on 22/12 17:49:56 1998 by HYPERCHEM
>
>    0   1
>  C,0,0.,0.,0.
>  C,1,1.5115
> [ ... stuff deleted here ...]
>  H,20,1.1137,5,111.4364,6,74.9555,0
>  H,22,0.9669,19,110.5133,21,-178.666,0
>
> --Link1--
> %Chk=abieticacid
> %NoSave
> #T B3LYP/6-31G(d) NMR Test Geom=AllCheck Guess=Read
> ----------------
>