Subject: LoBoS, a Beowulf cluster at NIH

From: Eric Billings <[email-masked]>
Date: Tue, 25 Nov 1997 12:32:53 -0500 (EST)

On Mon, 24 Nov 1997, Chris Harwell wrote:

> 	Are there any chemists out there who are using a beowulf cluster 
> of pcs type of system for computing? 
We have constructed an 80 processor Beowulf-class supercomputer called
LoBoS (Lots of Boxes on Shelves).  LoBoS will be scaled up to 136
processors next week.  LoBoS' home page is:
   http://www.lobos.nih.gov/
and contains a more complete description of the equipment.  We are in the
process of adding benchmarks for different parallel topologies.  BTW, the
hardware pulse and queue management code seen on the web page are freely
available.  As a government lab we can't put much vendor-specific info
on the web site (but we can talk about it).  It is safe to say that, for
our applications (QM and classical simulation) we are getting 10 times the 
price performance of the local supercomputer !

> 	Has anyone used gamess, gaussian, or Wien FLAPW on such a system? 
The test and current (Phase 2) systems have been in production
use since April performing CHARMM, GAMESS and AMBER runs.

>What hardware choices did you make (processor. motherboard etc.) ?
We tested the DEC Alpha, PowerPC, Pentium Pro and PentiumII processors.
The best price/performance was clearly (by a factor of 2-10) the Pentium
Pro.  The DEC Alpha remains an excellent processor so we are continuing to
develop a 5 processor DEC Alpha cluster.  Once we settled on the
PentiumPro we tested 5 motherboards and found a small variance (+/- 5%)
for well designed motherboards.  Cheap motherboards which were/are
available use the older Intel chipset and are 20-30% slower.

> 	What compiler did you use (gcc,f2c, g77)?
We tested 3 commercial compilers as well as f2c/gcc and g77.  We are using
the compiler from AbSoft.
 
> 	What OS/Distribution did you use (linux/ redhat or debian?)
All nodes use Linux.  I think its safe to say that the Beowulf community
is moving towards the use of RPM (RedHat) for installs and ensemble
management software distribution, but we have been using Debian with good
success.

> 	Any suggestions to someone considering building such a system?
1. At a recent conference there was a general consensus that the hardware
for up to 16 nodes was a "solved problem" when optimizing price
performance.  The equipment includes PentiumPros, a single Fast Ethernet
NIC and a 16-port Fast Ethernet Switch (not hub).  Add disk to suit your
taste.

2. Since it sounds like you're doing QM work, you might consider the Dual
PentiumPro configurations.  You can amortize the cost of supporting
hardware (disk, power supply, case etc) over two processors and improve
your overall price/performance.

3. Don't be taken in by the Intel Bunny people and automatically buy a
PentiumII.  The PentiumII is a PentiumPro with MMX (useless for
floating point ops) and faster memory access.  Unfortunately, the
advantages are offset by an off-chip cache that runs at half the speed
of the PentiumPro.  Our benchmarks showed that the PentiumII performance
improvements barely scale up with the faster clock rates.

- Eric

Eric Billings
Molecular Graphics and Simulation Laboratory
National Institutes of Health