Subject: Summary:Gaussian in PC clusters

From: Maija Lahtela <[email-masked]>
Date: Wed, 23 May 2001 06:40:57 +0300 (EEST)

Hi All,

Few days ago, I asked the question about Gaussian in PC clusters:

I have got several answers and useful hints. Thanks to All CCLers.
I may e-mail later to ask you more detailed questions.

Yours,
Maija Lahtela-Kakkonen

From: Szilveszter Juhos <[email-masked]>

Check the Condor job scheduling system at:

http://www.cs.wisc.edu/condor/

Furthermore be careful with the RedHat gcc compiler: it is not
reliable. If you can manage better to use Debian (for commercial support
check: www.progeny.com)

Szilva
***************************************************
From: [email-masked]

Dear Scientist
  if the problem of yor run is the speed so collecting many Pc May solve
it I like To Know How efficient the system become after that I think it
will be about 50 % of exepected
 but if the problem is Hard disk space the collection of many PC will
not solve this
 at last i like to know about your souce of Gaussian 98 under unix since
i Work with it under Windows98
And i wish a copy of answers you have recieved since i interested in the
cluster computer systema
*********************************************************
From: Dr. Robert Spiske <[email-masked]>

can you please give me a litle more detail to your gaussian?
We found that it takes 4 Mips R10000 @180MHz to overtake an 1.1MHz Athlon.
So i wonder what went wrong.

If you compile your gaussian with the normal settings it will be optimized
to
use 512 kb of cache. Modern Intel and AMD cpu's have only 256 kb cache.
Correct settings within the makefile can optimize the performace.

gaussian needs an enourmous amount of memory and lots for scratchfile
space.
therefor gaussian nodes should have a local swapfile (best) or a second
fast
nfs-network (not so good).
Some calculations (i believe it was CCSD) caused so much scratchfile
activity that the cpu-usage of the nfs-daemon was bigger than 50%. And 3
of
our 4 CPU's did nothing but waiting. A gaussian node should have at least
512MB/CPU if the memory is bigger the rest can serve as file cache
allowing
the system to continue faster after file i/o.

By the way there are problems with nfs within the linux 2.x.x-Kernels
that can cause severe problems!

To run gaussian in a cluster you will need the LINDA Extensions,
we did not do that by now. There are mpi informations within the code for
use
on HP-machines. Our university has a 32cpu version and we found gaussian
to scale up to 4-6 cpu sometimes 8. I dont think yo will not benefit by
the use of more cpu's in a single calculation.

Hope that helps a little?

head aega!

Robert
***************************************************
From: Doug Fox <[email-masked]>

  Maija,

   Carlos Sosa also copied me on some of this.  I did not see if you
responded to him with clarification on "slow execution" and in particular
"slow wrt to Windows" which is not our experience.  We use the same
compilers for Windows and Linux (Portland Group) and see very similar
performance.  So are these the same machines dual booted or at least
equivalent machines?  What types of calculations are you going for?

   The scaling for clusters will not go to 100 PC's, this is not a matter
of Linda vs. MPI.  Rather most problems are not large enough and so you
fall below the optimal size on each processor.  There are also issues
with communications overhead.  We support Linda for these clusters and
the licensing supports clusters of any size for G98+Linda so it is a
flat fee and you can run 12 8 processor calculations, this is done by a
number of groups.

   Also note that parallel execution works best for CPU bound processes
like HF, DFT and even CIS=DIRECT.  MP2 energy and gradients are also
reasonable because we can use local disk on each node to make disk I/O
and CPU in parallel.  If you are doing MP4, CBS-QB3, or G2 you are going
to hit disk I/O limits and not get great parallel speedups.  See the
paper in J. Comp. Chem. vol 19 1053, (1998) which gives you a good idea
what kind of speedups are likely for various methods.
********************************************************
From: John Bushnell <[email-masked]>

Hi,

  I saw your post on ccl.  We are also planning on setting up a
cluster for running Gaussian and other programs, and I am also
very interested in what has worked for others, and how well, etc.
I was curious what kinds of jobs you had run there for which
the performance was dissapointing.  Were they HF, DFT, or some
higher level of theory?  And were they "large" in terms of
number of basis sets?  I assume that you were
running Gaussian compiled with pgf on Linux?  I am
curious because most single cpu PC timings that I have seen
look pretty good, for instance the following recently reported
comparison (admittedly very small jobs!).  Specifically, the
relative speed (for these jobs) of the Pentium/Athlon systems
vs. an Origin 2000 (I am assuming an R10000?) seemed good:
**********************************************************
From: M. Nicklaus <[email-masked]>

For what it's worth:  Here's a recent small series of benchmarks we've run
on various Linux systems, plus a Cray and SGI Origin added for
comparison.

Program:        Gaussian 98 Rev. A.7.  All executables exactly identical
                for the Linux systems (copied between machines), compiled
                w/ PGI v. 3.2 (on Linux), G.98 standard compilation (no
                tuning/hacking).

Jobs run:       G.98 test jobs # 1, 28, 94, 155, 194, 296, 302, aggregate
                time, as reported in output, all runs single-CPU.

CPU   Speed  Chipset  Kernel   Distro   glibc    Memory    HD/Contr. time
                                               min. 256MB  MHz, RPM  (sec)
                                                         (all: U-ATA/)

P 4  1.5GHz Int. 850  2.4.2-2  RH 7.1   2.2.2-10 RDRAM-800 100, 7200 278
Athlon 1.33 VIA 686B  2.4.0   SuSE 7.1  2.2-7    DDR 266    66, 7200 299
P III 1 GHz Int. 815  2.4.3   (RH 6.1)  2.1.2    SDRAM 133 100, 7200 497
P III 1 GHz Int. 815  2.4.3   (RH 6.1)  2.1.2    SDRAM 133 100, 5400 497
P III 1 GHz Int. 815  2.2.12   RH 6.1   2.1.2    SDRAM 133 100, 7200 646
P III 1 GHz Int. 815  2.2.12   RH 6.1   2.1.2    SDRAM 133 100, 5400 646
P III 866MHz S/Works  2.4.3   (RH 6.2)  2.1.3-15 SDRAM 133  33, 7200 453
P III 866MHz S/Works  2.2.14   RH 6.2   2.1.3-15 SDRAM 133  33, 7200 526
SGI Origin 2000                                                      921
Cray SV-1                                                           2400

"(...)" in the Distro column denotes newer kernel than out-of-the-box.
Going from a 2.2 kernel to a 2.4 kernel seems to speed things up
considerably, at least on these systems.  Hard drive and IDE
controller speeds appear to be near irrelevant for these jobs

  Regarding the mpi version of Gaussian, I believe that is simply
a matter of compilation.  At least on SGI(IRIX) you can specify the
number of processors during compilation.  I assume that compilation
with the Portland Group compiler on Linux is similar.

  Yes, definitely summarize if you find out anything interesting.  I
think a lot of folks are considering PC clusters but facing a lot
of uncertainty as far as performance and implementation.

       - John              [email-masked]

PS:  Linda is expensive!  It would be nice to be able to use MPICH
if it was efficient.
**********************************************************
From: Christoph Steinbeck [email-masked]

I'm using a 16 node PC cluster for running Gaussian 98 using Linda as
the parallel execution environment. Works nicely.
It must, however, be noted that for our type of calculation, a
reasonable scaling is only achieved up to eight or ten nodes per job.
The results of our initial, provisional evaluation are summarized under
http://www.ice.mpg.de /departments/ChemInf/edda.html. Follow the last link
on the
scaling behaviour.

The result of this behaviour is that I use the OpenPBS batch queue
system to load-schedule jobs between subclusters of eight nodes each.

Cheers,

Chris

*****************************************************
Maija Lahtela-Kakkonen,
Application Scientist / Chemistry
CSC-Scientific Computing
Tekniikantie 15 a D, P.O.Box 405
FIN-02101 ESPOO  FINLAND
TEL 358-9-4572079 /050-3819506, FAX 358-9-4572302
E-MAIL [email-masked], Internet:www.csc.fi
****************************************************