Subject: Benchmarks and Software

From: David Young <[email-masked]>
Date: Thu, 22 Apr 1999 07:45:46 -0500 (CDT)

Hi,

	I would like to point out that this conversation on benchmarks is
a bit one dimensional.  There are some very popular (profitable) chemistry
software packages that do poorly on benchmarks but sell because of other
factors such as ease of use or functionality.  You really need to look at
a list of criteria when choosing a software package.  Some of them would
be

Benchmarks of efficiency

Cost

Ease of use in general

Suitability for a particular class of chemical systems.

Robustness - what percentage of the time does the job fail to execute even
though the inputs are in accordance with the manual?

Functionality - does it compute the properties you need?

	Once you have this information, you still might get a program that
does very poorly on several of the criteria in order to address the
criteria that is your highest priority.  Many people classify ease of use
as the one they have used before thus preserving their five year
investment of previous time rather than investing another 2 weeks to learn
to use another program that would be considered much easier to use.

	That said, now consider benchmarks specifically.  The best model
to follow would probably be the one used for CPU benchmarks.  These fall
into three categories.

1.  Benchmarks done by an independent organization.
2.  Benchmarks that require that the code be unaltered and compiled
without options.
3.  Benchmarks in which almost anything is fair game as long as the final
answer is correct.

In the case of chemistry software, you have to worry about the chemical
systems in the benchmark set.  A program that does very well on a set of
organic molecules may perform miserably or even completely fail to run
with a set of transition metals.  Computer hardware is also a big factor. 
One program may do better on PCs while another does better on machines
with vector math units.  Ideally you would want a break down of all of
these factors and one big number at the end. 

	Who does the benchmarks is also a question.  Academics may not
have a suitable selection of hardware to test codes on and may not wish to
publish poor results for fear that the writer of the codes may be a future
reviewer.  National labs are becoming more tightly tied to industry, thus
creating political problems.  Industrial workers don't want to spend so
much effort unless there is a way to profit from it.  Ideally an
independent organization (perhaps privately held company) would get
hardware manufacturers to give them hardware and charge them a fee besides
to do the benchmarks, then they would charge each software company a fee
to do their benchemarks, then the benchmarks would be published or on the
web free.  Perhaps a company that already benchmarks CPUs should hire a
chemist to develop and run these tests on the same machines.  I wonder if
they could get enough money from the project to pay the chemist and add to
profits besides?  Perhaps the fee they charge should be based on the
selling price of the software.

                                Dave Young
                                [email-masked]

------------------------------------------------------------------------

          Live long and maintain marketable job skills.

------------------------------------------------------------------------