Subject: Summary- "Best" Multiproces

From: "John Beckerle" <[email-masked]>
Date: 15 Apr 1997 11:11:34 -0500

Here is the belated summary of the responses to my query about the "best"
multiprocessor platform for a broad range of computational chemistry
applications.  The executive summary is that under real world circumstances,
the respondents were happiest with the offerings from SGI, i.e. the
PowerChallenge.  Thanks to all who responded.  I have included in the summary
some of the give and take I had with the responders.  

My original question:

> We are seriously considering investing in a multiprocessor compute server.
> The candidates include the IBM SP2, SGI Origin2000, Sun Ultra HPC, DEC
> Alphaserver and other similar machines.  We are probably looking at
something
> on the order of 8 processors and 2GB of memory, although expandability and
> upgradability are important.  The system is supposed to serve a variety of
> engineering and scientific high performance computing needs, not just
> computational chemistry.
> 
> I would like to poll the collective experience of CCL as to what is the best
> platform, and even more importantly, which platforms should be avoided for
> computational chemistry applications.  I am interested in hearing about real
> experience with software availability, incompatibilities, and performance
> surprises (including effective use of parallelism), both good and bad.  We
are
> interested in the full range of computational chemistry applications
including
> SCF and MD. 
> 
> I would like to stress that I am looking for good and bad experiences with
> effectively using any of these multiprocessor machines and the roadblocks
> others have encountered.  I am not so interested in relatively minor
> differences in performance of the individual CPUs, benchmarks, etc.  I want
to
> know if they can be used effectively without miracles.
> 
> I am already familiar with the extensive evaluation of workstation
performance
> by Martyn Guest, as well as Milan Hodoscek's comparison for parallel MD
> speedups.
> 
> As I don't subscribe to CCL at this time, please respond directly to:
> [email-masked]
===============================================================
From Dr. Eric V. Patterson:
I have experience running a variety of computational chemistry programs 
(Gaussian, MOLCAS, ACES II, etc.) on both an IBM SP2 system and an SGI 
Power Challenge, the forerunner to the Origin2000.  Forgive me if I'm 
covering familiar ground, but the main difference between these machines 
is the way they handle parallel operation.  The SP2 is series of 
individual nodes (e.g. RS/6000 machine "guts") with individual disk and 
memory (although the disk can be shared).  The nodes communicate via a very 
fast network.  The SGI is a shared memory machine, where the processors 
access the same the disks and same memory, and each processor is just a 
different chip on a card.

I prefer the SGI.  The basic reason is that it is one single machine with 
several processors.  The processors have few problems communicating with 
each other, and it is easier to run a standard queueing system such as 
NQS.  With the SP2, you must run LoadLeveler as your queueing system, 
which presents a unique set of hardships.  Plus, in the case of Gaussian 
at least, you need different Fortran libraries (LINDA) in order for the 
compiled program to work with LoadLeveler.  This may be true for other 
programs as well.

As far as other factors go, both machines are good performers.  If a 
program, chemical or otherwise, has been well written, it will 
parallelize well on either machine.  Also, both are fairly easily 
expanded.  For the SP2, you add more nodes until your rack is full, 
then can buy a new rack.  For the SGI, you can add more processor 
cards until you have no more space.  Adding disk and memory is no more 
difficult.  So, my preference for the SGI is simply from an "ease of use" 
consideration.

I hope this helps.  

Beckerle responds:
> I know G94 can be set up to run on the SP2 with Linda, and that Gaussian
> supports that configuration.  What about other software packages?  

Patterson:
The other packages I'm familiar with on the SP2 are GAMESS and MOLCAS.  
GAMESS is coded to handle message passing in a different way (I don't 
recall which way), so it does not require Linda.  It performs fairly well 
in parallel up to about 8 nodes.  MOLCAS does not run in parallel.  I am not 
familiar enough with any other scientific applications to know what they are 
capable of.

Beckerle:
>In
> practice, have you found that there are straightforward instructions and/or
> support to get things running on the SP2?  Or, do you tend to just run most
> calculations on the PowerChallenge instead? Another good measure:  How much
> idle time do you have on the SP2 as compared to the PowerChallenge? 

Patterson:
Once LoadLeveler is running, using the SP2 is no more difficult than any 
other queue-based machine.  However, I feel that LoadLeveler is less 
stable.  I think the reason for this is that you are dealing, in essence, 
with a cluster of networked machines.  I have had jobs crash, or seen the 
whole queueing system die, due to interuptions in the network.  The 
common examples occur when the LoadLeveler master node becomes 
unreachable, or when the disk that contains the program code become 
unreachable.  These problems happen a little too often for my taste.  
Because of that, jobs on the SP2 are often run in serial mode, and the 
SP2 remains more idle than the SGI.  

Getting LoadLevel running seems to be a stumbling block for many people.  
I have never had to set it up from scratch, but I have talked with the 
people who have.  It seems to be a chore.  I have set up a new SGI, and 
it was straightforward.  The OS on the SGI handles all of the parallel 
issues transparently, while the SP2 requires the operator to install 
message passing tools.  So, if you have ever set up a typical 
workstation, you can probably set up the Origin system with little 
effort.  Even if you haven't set up a workstation before, the SGI 
interface is a bit nicer than IBM's, so it tends to be more comfortable 
for people.

If you have other questions, or I haven't addressed certain key points, 
let me know.  I'm happy to share my experiences.

Eric
Dr. Eric V. Patterson		
Postdoctoral Associate 

Department of Chemistry		voice:	(612) 624-1535
University of Minnesota		FAX:	(612) 626-7541
207 Pleasant St. SE		email:	[email-masked]
Minneapolis, MN 55455		WWW:	http://pollux.chem.umn.edu/~patter

================================================================
From Frank Herrmann:
I personally don't like the SP2 philosophy. You have to login on every
single processor or use a queueing system to have access to the whole
machine. I prefer shared memory machines like the Convex/HP SPP1 or
SGI Power Challenge. You don't have to bother on which Processor you
are or whether some of them are down or which processor is already
busy and so on - It's like being on a single processor machine. The
Convex and SGI are also much faster at least with my program (C
Programm with very little memory consumption and low input/output
activity - not the typical Gaussian job properties) Especially the SGI
Power Challenge with R10000 Processors is more than 5 times faster
than the SP2. But maybe there are faster SP2's because the R10000
machine is brandnew ...

---------------------------------------------------------------------
           Frank Herrmann, Computer Scientist, PhD Student            
   Institute of Parallel and Distributed High-Performance Systems    
                   (IPVR) University of Stuttgart                     
                      Breitwiesenstrasse 20-22                        
                    D-70565 Stuttgart  (Germany)                      
           Tel: (49) 711-7816-358, FAX: (49) 711-7816-250            
          email: [email-masked]         
http://www.informatik.uni-stuttgart.de/ipvr/bv/personen/herrmann.html

=================================================================
From Dr. Jack Miller:
We have had very goood luck and very good performance from multirocessor
SGI systems starting back with the old 340's then with the Challenge series
and are just preparing to by one of the new Origin systems. They have been
real workhorses and were the bast bang for the buck that ran our software
requirements at the time we went with SGI six years ago.

Six years ago IBM could win on a single job benchmark, but as soon as we
went to multiple jobs SGI won hands down --- the OS was very much more
efficient at swapping jobs to share the multiple  processors where the IBM
threshed about in a multi-job environment. We used our own programs that
our theoreticians wrote for a Cray and various packages such as Gaussian
that we knew we would have in our job mix. The results on a real job mix
were very different than the raw performance figures based on SPECmarks etc
would have suggested.

Jack Martin Miller
Professor of Chemistry
Adjunct Professor of Computer Science
Brock University,
St. Catharines, Ontario, Canada, L2S 3A1.

Phone (905) 688 5550, ext 3402
FAX   (905) 682 9020
e-mail [email-masked]
http://chemiris.labs.brocku.ca/~chemweb/faculty/miller/
===================================================================
From Rick Venable:
Several groups have projects well underway to assemble their own parallel
systems using "off the shelf" components from the PC marketplace:

dual processor PentiumPro w. motherboard
128 MB RAM, 2 GB disk
multiport 100baseT fast ethernet
case, power supply, keyboard, monitor, display adapter
Linux
etc.

The commodity status of PC components and recent performance gains means
one can shop around a bit, and assemble a dedicated parallel system for
ca. $20,000 that outperforms commercial multiprocessor machines costing 5
to 10 times more.  Don't buy a service contract, just some extra parts. 
Of course, this approach does require one or two more technical
individuals on the scene, but it also represents a radical
price/performace breakthrough.  A group here at NIH is developing one such
system, and will eventually have a WWW page with a shopping list and a
recipe.  Milan's parallel CHARMM timings should now include results for
one and two processor PentiumPro machines.

At NIH, they call the project LoBoS (Lots of Boxes on Shelves); the name
is also an homage to early innovaters in this area, who coined the term
"beowulf" to describe the general concept.  For more information, contact
the project leader, Eric Billings at [email-masked]

On 5 Apr 1997, John Beckerle wrote:
> But is this case I don't want to turn the hardware and software into the
> research project.  This system is supposed to support a variety of users and
> applications from different departments.  If you consider the real cost of
the
> expertise required to set up and maintain a system and to port software to
it,
> and debug the system stuff, and rewrite the code to get better parallelism,
> and ....  Well it adds up to a lot of personnel cost and time.   Before too
> long, you will be able to pick up the ready to run parallelized software for
> whatever you want to do from the same place as the list of components for
the
> hardware.  But I would still rather pay an expert to set it all up.  

I don't think the expertise required will be much beyond that needed by
any competent Unix sysadmin, especially someone who can add memory or
replace a disk, and handle TCP/IP configuration.  You should be able to
leverage the results of groups like the one at NIH so that it is not a
research project.  With regard to performance tuning, in my experience
that's often necessary for any change of platform, especially
multiprocessor systems, and in some cases a change of OS.  For any
parallel codes that use widely available parallel libraries such as PVM, I
expect very little effort would be required.

If your need for a multiprocessor system is more to support lots of
single-threaded jobs, and/or you have mostly code which is not explicitly
parallelized and require a compiler which will multi-thread compute
intensive inner loops, then I'd advise against the self-assembled parallel
PentiumPro approach.  For more dedicated use, e.g. only computational
chemistry with packages such as CHARMM and/or GAMESS, however, the low
cost will give groups who adopt this approach a major advantage over their
scientific rivals in terms of computational power. 

Rick Venable                  =====\     |=|    "Eschew Obfuscation"
FDA/CBER Biophysics Lab       |____/     |=|
Bethesda, MD  U.S.A.          |   \    / |=|  ( Not an official statement or
[email-masked]  |    \  /  |=|    position of the FDA; for that,
http://nmr1.cber.nih.gov/           \/   |=|    see   http://www.fda.gov  )
======================================================================