Subject: SUMMARY of Beowulf for chemistry

From: Chris Harwell <[email-masked]>
Date: Thu, 4 Dec 1997 10:45:07 -0600 (CST)

hello,

thanks very much to all who responded to my query about bewulf being used 
in chemsitry!

Beowulf Systems in Computation Chemistry

can also be found at:
http://www.chem.lsu.edu/htdocs/people/crharwell/mybeo.htmlB

These are in responce to a CCL and GU queries for information on beowulf systems in application to computational chemistry.

We have been pondering the same thing, so if you get any
feedback please let me know!

I think that in terms of OS distributions it probably
doesn't matter; what _is_ important is to get the
interprocessor communications software from the Beowulf/Loki
sites/home pages. GAMESS uses TCGMSG (if my failing memory)
is correct, and so it will have to be reconciled with
whatever is being used on Beowulf.

=======================================================================
Craig W. Burkhart                   [email-masked]
Goodyear Research
=======================================================================

> 	Has anyone compiled and run gamess on a PC/Linux cluster?
We've done this using FreeBSD (<A href="http://www.freebsd.org"> FreeBSD </A> ), as both
Linux and BSD use the same tools and programming interfaces
you should be able to do the same thing.

> 	If so, with what amount of success?

Very good, we started some years ago with 486 machines an
now we're running on PPro 2000.

> 	Any benchmarks or ballpark times?

I'm CC: to [email-masked]. Dr. Nelson has the
most recent results.

> 	What hardware? (processor, motherboard, ethernet?/fast/myri/atm, mem)

ASUSTEK MBoard with PPro200/256k cache, 64M RAM, NCR SCSI2 controller
and Seagate HDs. We use PCI Fast Ethernet Cards based on DEC DE21140
chips or Intel Pro 10/100B these machines are attached to a Hub
and to a Xylan switch. We have plans to move to faster network
technologies in the next year.

> 	What compiler? (gcc f2c or g77?)

Both compiler produce good results, in true we're using a
mix, some GAMESS's modules produce faster code with f2c others
are faster under g77 (we're running 0.19/0.20)

> 	What OS/distirbution? (linux/ redhat or debian?)

We're runing this for the last 3 years or so, we started
with freebsd 1.1.5 and now we're using  2.2.5. You should
have no problems runing redhat/debian/slackware either.
Pedro

Hello,

I built a Linux cluster some time ago, you can get answers to 
most of your questions by having a look at our cluster's webpage:\
    <A href="http://bigbrother.ag1.mpi-sb.mpg.de"> bigbrother </A>
I compiled GAMESS using g77 and it works fine. The problem with gamess is,
it doesn't scale very well with the number of processors.
When executing bench11.inp of the GAMESS test suite, I got a spedup of about
4, using 12-16 processors. This seems to be mainly because of GAMESS being 
only partially parallelized and also becaus of high latency times and low 
bandwith in our 100Mbps network.
The distribution we installed was Slackware (Linux 2.0.25), but I would prefer 
Debian now.

 Oliver Kohlbacher   ([email-masked])
 Max-Planck-Institut fuer Informatik, Im Stadtwald, 66121 Saarbruecken
 Tel.: 0681-9325-505 Fax: 0681-9325-199

Hallo Chris,
I don't know anything about this special cluster, and if you want to
realize an cluster for parallel processing, then I have no experience.
However, we have running a little test cluster for quantum chemical
calculations also under Linux. We have running most of the known
ab initio quantum chemical software packages including GAMESS (see
also <href="http://www.chm.tu-dresden.de/organik/qc/linux.html"> here </A> ).
The most important condition for good performance is an batch system;
we are using the free available Generic NQS. The software file system
is shared in the cluster via NFS. Up to date, there are no problems
and we are using this machines also for production work. The most
important limit in comparison to workstations is the main memory
(we have 64 MByte on the PC's) and the memory cache. The performance
is good for smaller jobs and goes back with jobs near the memory limit
(see also the above mentioned web page, there are links to some 
benchmarks we have done with GAMESS under Linux, IBM RS6000 and
Origin2000).
Our hardware is Pentium PC (166 and 200) with 64 MB Memory, 4 GByte disk 
and Adaptec SCSI interface. We are using only the old BNC ethernet cable 
(without problems, but you should be sure to save the temporary integral 
files on a local disk and not via NFS).
In the special case of GAMESS I have used the g77 compiler because of some
error messages with f2c/gcc. Our Linux comes from a German distributor
(SuSE GmbH), but I'm sure, that other distributions will work also.
Hope this information will help !
Matthias

Dr. Matthias Mann
Computational Chemistry Group
TU Dresden, Germany
[email-masked]

Chris,

Iwould dearly love to hear of any responses you get to this question.  I am
trying to build such a system at th moment.  First we are trying to build a
prototype system (to see if we can do it) and then expand it (maybe).

WHat we are looking at at the moment is (hopefully all free :-));

486 DX2/66, PCI motherboard, 16MB, 500MB, 100Mbit ethernet.

Then, if we can get this working at a reasonable speed, we might start to
swap into updated hardware.  Our first problem is to get free hardware, and
work up from there, but if someone can show Gaussian / Gamess working, then
it might be worth diving in at the deep-end.

ALso, you might like to take a look at PAPERS as a connection medium (see
<A href="http://garage.ecn.purdue.edu/~papers/"> papers </A> for more info).  ALso at that site is
a discussion of the different networking methodologies that are best (ATM/
Ethernet/ PAPERS / whatever).

Good luck, and please let me know how you get on.

--	Simon Hogg,
	Imperial College, London, UK

>
>	Are there any chemists out there who are using a beowulf cluster 
>of pcs type of system for computing? 

I haven't personally, but I know Morokuma's group at Emory does.   A
cluster of 14 Pentium Pro 200 PCs, I believe.

Jaime Martell, Ph.D.			P.O. Box 2317
Camille and Henry Dreyfus Fellow	Batesville, AR 72503-2317
Biology and Chemistry Division	Phone:  870-698-4688	
Lyon College				Fax:  870-698-4622

>I compiled GAMESS using g77 and it works fine. The problem with gamess is,
>it doesn't scale very well with the number of processors.
>When executing bench11.inp of the GAMESS test suite, I got a spedup of about
>4, using 12-16 processors. This seems to be mainly because of GAMESS being
>only partially parallelized and also becaus of high latency times and low
>bandwith in our 100Mbps network.

1.	Here at Ameslab, Scalable Computing Lab
<A href="http://www.scl.ameslab.gov"> ameslab </A> found that they had trouble getting faster
than 1Mbps on Switched fast ethernet under linux for some packet sizes.
Linux's tcp/ip stack and network drivers were really bad.  Downloading the
latest drivers for you ethernet card, and the latest version of linux might
help.  Performance was packet size dependent.  For one code, you could
increase the size of the problem, and thus the packet size, and have it
finish much faster.

2.	Bench11.inp is not big at all.  You would never run such a job in
parallel.  Small parallel jobs sometimes take longer on multiple nodes,
just because of all the "I'm done, are you done" talking.  It only takes
18.6 CPU minutes (28 minutes Wall) on one of our 266MHz Alpha's.

3.	CI is not the most scalable part of the code either.

Kurt Glaesemann
201 Spedding Hall
Ames Lab/Iowa State University
Ames, IA 50011
________________________________

Yes.  Check out <A href="http://www.lobos.nih.gov"> lobos </A>.
------------------------------------------------------------------------
 Marc C. Nicklaus                        National Institutes of Health
 E-mail: [email-masked]               Bldg 37, Rm 5B29
 Phone:  (301) 402-3111                  BETHESDA, MD 20892-4255    USA
 Fax:    (301) 496-5839    
      <A href="http://rex.nci.nih.gov/RESEARCH/basic/medchem/mcnbio.htm "> web </A>
    Laboratory of Medicinal Chemistry, National Cancer Institute,  &
  Lab. of Structural Biology, Div. of Computer Research and Technology
------------------------------------------------------------------------

Dear Chris,

  we are indeed thinking about building a beowulf cluster (like many
other groups I guess). In particular we are considering a 16 Pentium
Pro 200 machine, connected by means of a fast ethernet switch.

>From the point of view of the hardware we are still evaluating the
feasibility of our project. We are testing MPI on a 3 Pentium 166 MMX,
ethernet connected system. Obviously, we will not obtain significant
informations about performance from that system, but I must say
that everything work nicely! (btw we used the RedHat 4.1 or 4.2 linux
distributions)

About the software. We developed a modified version of Gaussian94
that uses MPI as message passing library. It also use ScaLAPACK's
parallel linear algebra routines for diagonalization and matrix
multiplication.
We developed that code on the Cray-T3E MPP system where we obtained
good results (I can send you some data if you want).
We ported the basic MPI inteface on our toy-PC-cluster and it works,
but we do not yet know the actual performance of the ScaLAPACK library 
routines within G94. To use ScaLAPACK you have to scatter/gather the
relevant matrices from/to the master and the workers processes and
this imply a lot of communication overhead. If it's true that
communications are **VERY** fast on the Cray-T3E, we are not so sure
about the actual performance of MPI-based communications in a PCs
cluster!

We are very interested in testing our Gaussian94-MPI-ScaLAPACK code
on a true [ already existing :-) ] Beowulf system, so I will be glad
if you could copy me all the answers you obtain from the CCL (or I'll
wait for a summary!)

----------------------------------------------------------------------
     | SCALMANI Giovanni                 [email-masked]
 ^^^ | Ph.D. Student
 o o | Dipartimento di Chimica Fisica ed Elettrochimica
  |  | Universita' degli Studi di Milano
 \_/ | via C.Golgi, 19                          Phone: ++39-2-26603254
     | 20133 Milano (Italy)                     Fax  : ++39-2-70638129
----------------------------------------------------------------------

Hi,

> 	Are you aware of anyone one using wien in parrallel on pc clusters? 
No.

> If so do you know of any benchmarking/ performace testing ie how 
> parallelizable is wien? 

If you do k-point parallelization (WIEN97) it should run with full speed of 
each processor. Of course it depends on the cases you want to investigate 
if you have enough k-points to distribute.

> 
> Any suggestions on 
>  hardware configuration(proc, motherboard,mem, hardisk, fastether/atm/giga?)
>  software (compilers gcc vs g77?, message passsing,load balancing)
>  or os distribution (redhat or debian)?

We use a SUSE distribution and g77 on a PII-266MHz. It has the speed of about
1/3-1/4 of a state-of-the-art SGI R10000, DEC Alpha-500MHz or IBM 120MHz 
processor. (The new HP or IBM 375-160MHz will be even faster).
With k-point parallization you need just standard ethernet and a common
NFS filesystem, no extra hardware, message passign software,...

However, you need "full" memory on each node (probably at least 128 Mb, maybe
even 512, depending on your needs).

--------------------------------------------------------------------------
Peter BLAHA, Inst.f.Techn.Elektrochemie, TU Vienna, A-1060 Vienna
Phone: +43-1-58801-5187             FAX: +43-1-5868937
Email: [email-masked]    WWW: <A href="http://www.tuwien.ac.at/theochem/"> web </A>
--------------------------------------------------------------------------

> 	Are there any chemists out there who are using a beowulf cluster 
> of pcs type of system for computing? 
We have constructed an 80 processor Beowulf-class supercomputer called
LoBoS (Lots of Boxes on Shelves).  LoBoS will be scaled up to 136
processors next week.  LoBoS' home page is:
   <A href="http://www.lobos.nih.gov"> lobos </A>
and contains a more complete description of the equipment.  We are in the
process of adding benchmarks for different parallel topologies.  BTW, the
hardware pulse and queue management code seen on the web page are freely
available.  As a government lab we can't put much vendor-specific info
on the web site (but we can talk about it).  It is safe to say that, for
our applications (QM and classical simulation) we are getting 10 times the 
price performance of the local supercomputer !

> 	Has anyone used gamess, gaussian, or Wien FLAPW on such a system? 
The test and current (Phase 2) systems have been in production
use since April performing CHARMM, GAMESS and AMBER runs.

>What hardware choices did you make (processor. motherboard etc.) ?
We tested the DEC Alpha, PowerPC, Pentium Pro and PentiumII processors.
The best price/performance was clearly (by a factor of 2-10) the Pentium
Pro.  The DEC Alpha remains an excellent processor so we are continuing to
develop a 5 processor DEC Alpha cluster.  Once we settled on the
PentiumPro we tested 5 motherboards and found a small variance (+/- 5%)
for well designed motherboards.  Cheap motherboards which were/are
available use the older Intel chipset and are 20-30% slower.

> 	What compiler did you use (gcc,f2c, g77)?
We tested 3 commercial compilers as well as f2c/gcc and g77.  We are using
the compiler from AbSoft.
 
> 	What OS/Distribution did you use (linux/ redhat or debian?)
All nodes use Linux.  I think its safe to say that the Beowulf community
is moving towards the use of RPM (RedHat) for installs and ensemble
management software distribution, but we have been using Debian with good
success.

> 	Any suggestions to someone considering building such a system?
1. At a recent conference there was a general consensus that the hardware
for up to 16 nodes was a "solved problem" when optimizing price
performance.  The equipment includes PentiumPros, a single Fast Ethernet
NIC and a 16-port Fast Ethernet Switch (not hub).  Add disk to suit your
taste.

2. Since it sounds like you're doing QM work, you might consider the Dual
PentiumPro configurations.  You can amortize the cost of supporting
hardware (disk, power supply, case etc) over two processors and improve
your overall price/performance.

3. Don't be taken in by the Intel Bunny people and automatically buy a
PentiumII.  The PentiumII is a PentiumPro with MMX (useless for
floating point ops) and faster memory access.  Unfortunately, the
advantages are offset by an off-chip cache that runs at half the speed
of the PentiumPro.  Our benchmarks showed that the PentiumII performance
improvements barely scale up with the faster clock rates.

Eric Billings
Molecular Graphics and Simulation Laboratory
National Institutes of Health

>...
>3. Don't be taken in by the Intel Bunny people and automatically buy a
>PentiumII.  The PentiumII is a PentiumPro with MMX (useless for
>floating point ops) and faster memory access.  Unfortunately, the
>advantages are offset by an off-chip cache that runs at half the speed
>of the PentiumPro.  Our benchmarks showed that the PentiumII performance
>improvements barely scale up with the faster clock rates.

Another issue is the cost of processor in the future: PPro's architecture
(on-chip cache) makes it very costly in production. This is another reason
Intel came up with PentiumII with off-chip cache. The off-chip cache structure
is also cheaper to expand, both in terms of size and clock speed.

So bottom line is that PPro with on-chip cache has a better architecture
than Pentium II for computional extensive work, but Intel might not lower
the price further or devepop it more (high clock speed or more cache).  The
cost/performance ration of two CPU would change in the future.

Xinjun

C     Xinjun J. Hou ([email-masked])      Agouron Pharmaceuticals, Inc.