Subject: Benchmark summary

From: Charles Letner <[email-masked]>
Date: Fri, 4 Aug 1995 13:13:13 -0400 (EDT)

Hello all,
	Two weeks ago I possed the question about benchmarks.  As promixed
here is the summary.
Thanks to all who responded!
Chuck 

Charles Letner
Wright State University
Department of Biochemistry
Dayton, OH 45435
e-mail: [email-masked]

Original post:

Hello everyone,
	I would like to start adding some basic benchmarking to some code
I have (both C/C++ and fortran).  But I have a couple of question.  First,
what are your opinions on various benchmarks.  For example, cpu time could
be one benchmark.  However I see this as problematic.  I'm working on
moving code to parallel machines.  CPU time doesn't appear to me to take
into account that the code could potentially take up more cpu time due to
overhead involved in running multipule cpus while executing much faster in
terms of "wall clock" time, ie/ user time.  Mflops seems like an
alternative that would overcome this but may be harder to implement.
	This brings me to my second question.  How are you implementing
these methods into your code?  I'd be interested in function calls,
libraries, etc... that could be used at runtime.  I would like
to have methods that are portable across different plateforms as the
ultimate idea is to compare benchmarks of actual runs on multipule
plateforms.  I would also be interested in any references on this topic.
	I sincerely appretiate any time you take in responding.  I will
summarize any responses to the net towards the end of next week.
Regards,
Chuck

Charles Letner
Wright State University
Department of Biochemistry
Dayton, OH 45435
e-mail: [email-masked]

Responses:

From: Hinsen Konrad <[email-masked]>

Well, you should calculate whatever is relevant for you. CPU time
is relevant if you are billed for it, and wall clock time is
relevant because you have to wait for it. So why not have both?
MFLOPS are pretty useless; they express to speed of a piece
of hardware, and that with reference to hardware-dependent
units.

I doubt that there is anything really portable for parallel machines.
But every operating system provides some functions for finding the CPU
time; it should be in the documentation. IRIX, like many other Unix
versions, has the system function times() to be called from C and C++
programs and the Fortran functions etime() and dtime().  I wouldn't
worry too much about portability - just write a single
machine-dependent function to calculate what you want; it won't be
more than a couple of lines.

-------------------------------------------------------------------------------
Konrad Hinsen                     | E-Mail: [email-masked]
Departement de chimie             | Tel.: +1-514-343-6111 ext. 3953
Universite de Montreal            | Fax:  +1-514-343-7586
C.P. 6128, succ. A                | Deutsch/Esperanto/English/Nederlands/
Montreal (QC) H3C 3J7             | Francais (phase experimentale)
-------------------------------------------------------------------------------

From: Joe Leonard <[email-masked]>

Chuck - can you use time-to-solution as a benchmark?  This lets you factor in
things like workstation clusters and parallel machines, etc, and dispenses with
individual timing on routines...

If you have a set of molecules you want to perform calculations on, can you use
the time it takes to carry out the calculations in various computing
enviromments.  This way, you can tune individual pieces of code however the
vendor suggests and get a consistent measure.

Might be full of it, but we use it here when measuring the efficiency of things
like conformational analysis when multi-machine enviromnents are used...

Joe

------------------------------------------------------------------------
Joe Leonard
Wavefunction Inc.
18401 Von Karman, Suite 370
Irvine, CA  92715                       I am a professional...
714-955-2120                                    do not attempt this at home.
714-955-2118 fax
[email-masked]

From: "Timothy G. Mattson" <[email-masked]>

Chuck,

You want to use Wall time.  CPU time is a poor choice since it doesn't
always include time spent inside the OS on behalf of the benchmark.
For example, CPU time doesn't pick up time spent managing ethernet
conflicts.  

MFLOPS only works as a performance metric when you have a mathematical
definition of the operation counts.  For example, in counting flops,
how many flop's do you assign to a square root.  The actual count varries
>from one machine to another making MFLOPS a non-uniform metric.

The only way to go is wall time.

In terms of portable parallel computing, you have many options. 
A popular choice is TCGMSG.  This is a small and easy to learn
message passing environment designed by and for chemists.  You
also could go with PVM or MPI.   If you want to learn more about
portable parallel programming environments, take a look at my
paper on the topic  which you can grab by anonymous ftp

   export.ssd.intel.com
   pub/tmp/mattson/ijsa.ps

This is a temporary location, so grab the paper right away if
you are interested.

--Tim

--------------------------------------------------------------------
Timothy G. Mattson, Ph.D. - Senior Scientist 

 Intel Supercomputer Systems Division  | Phone: (503) 677-5627
 Mail Stop CO1-15                      | FAX:   (503) 677-6324
 5200 N.E. ELAM Young Pkwy             | email: [email-masked]
 Hillsboro, OR 97124-6497

--------------------------------------------------------------------
  
From: [email-masked]

Parallel machines typically dedicate nodes to a job, so
wallclock time is a 1st approximation, although traffic
between nodes from other jobs could have an impact.

The fortran cpu & wallclock time routines from Amber 
(along w/ the compiling system) are freely available
for use & redistribution, if you want a wrapper to
hide system dependencies. If you're interested, I'll
make up a package w/ the relevant stuff.

Bill Ross

[The package is at]

The package is in anon ftp on odin.ucsf.edu under amber/CompSys.tar.Z

Bill

From: Jeyapandian Kottalam <[email-masked]>

On all Cray platforms we use wallclock times (call secondr or rtf).

On vector-parallel platform (traditional ymp type) wallclock times
are meaningful only when the machine is dedicated for the benchmark
job.  This is indeed how we run benchmarks, i.e., by taking the machine
down and bringing it up accesible only to selected users.

On massively parallel system (Cray T3D) the benchmarking job can
run during batch time to get a pretty good measurement.  This is
because when a job starts running a certain number of processors
get dedicated to that job.  But the communication network is being
shared with other jobs and that can make the measurement slightly
inaccurate.

Kottalam                           Phone: (USA) 612 683 3622
Cray Research, Inc.

From: Matt Clark <[email-masked]>

Regarding what measurement to use in a benchmark, I feel you should
measure what you are really interested in. Namely "how long does it take
to run", or wall clock time. If you use CPU time you would add up the
time for each individual CPU, and that isn't a very useful measure of the
performance increase of a parallel computer. Wall clock time takes into
account any paging/swapping/io that the program causes, which are 
important practical issues.

From: Peter Shenkin <[email-masked]>

> From [email-masked]  Fri Jul 21 14:21:18 1995

> ...Most [responses] indicate that "wall clock" time
> is probably one of the better benchmarks.  The most common reason: it
> takes into account issues such as multipule processors, memory swapping,
> etc..  To eliminate factors such as other jobs and other users one
> indiviual mentioned that they make the machine unavailable to other users
> while the benchmark job is running.

I must disagree with this conclusion.  The problem is that benchmarks
run on different machines without kicking off the other users will
inevitably be run with different loadings on the different machines.  
If machine X appears faster than machine Y, you don't know whether 
it's because machine Y had 10 users running big jobs whereas X only 
had 2 users running big jobs when the benchmarks were run.  This can
dramatically effect things like swapping, which increases wall-clock
time greatly.

CPU time is still the best benchmark, IMHO.  It is sufficiently 
insensitive to loading to be able to be fairly run on a loaded machine.
Parallel code presents additional difficulties, admittedly, but
the sum of the CPU times does in fact tell you a lot, especially
when compared to the CPU time when the same job is run on a 
single processor.

	-P.

***************** Atheist:  An evangelical agnostic. *****************
*Peter S. Shenkin, Box 768 Havemeyer Hall, Chemistry, Columbia Univ.,*
*NY, NY  10027;  [email-masked];  (212)854-5143;  FAX: 678-9039*
********** World music:  What Bluegrass is to a Bulgarian. ***********

From: Swamy Kandadai <[email-masked]>

Charlie:

The issues you are raising are valid and I am sure you will get several
answers. What we (in performance analysis) is really interested is in
finding out what machines best perform for a special application.
If the application is cpu intensive (eg. most molecular dynamics codes
assuming you don't save every coordinate and velocity at every step of
the molecular dynamics step),  then cpu  is an important characteristic
in a serial run. On the otherhand, if the application involves some
amount of I/O as you normally see in traditional scf calculations
(writing 2-e integrals and reading them at every iteration step to form
the Fock matrix), the elapsed time is important in addition to cpu
time since the ratio of elapsed to cpu will tell us the I/O bandwidth
on that particular machine.

In parallel applications, what we are interested is fast turn around time.
Everybody understands taht running parallel job on a loaded system is not
the best way to run a parallel job. In parallel applications, cpu time is
misleading since different cpu times indicates probalble bottelnecks in
load balancing. In message-passing applications, it will also tell about
how much time the application is spending on the messages to happen which
in turn will tell us about the bandwidth of the medium of transfer.

A single measure as cpu or the elapsed time is not going to be sufficient
and the application characteristics will dictate what to look for.

Thanks
Swamy Kandadai