Subject: Re: CCL:Benchmarks and software

From: Don Gregory <[email-masked]>
Date: Wed, 21 Apr 1999 10:40:21 -0700

Warning:  short, perhaps controversial response follows.
; )

>>On Mon, 19 Apr 1999, David Giesen wrote:
>> I would like to add a second requirement of all commercial software if I
>> may dream a little.  

> While Peter Shenkin replied .....
>Where there's life, there's hope. :-)

A comment on a related statement made earlier, plus a small additional
note of my own.

It was pointed out that it is *very* difficult to get consensus from any
significant number of people, on what 'banchmarks' are acceptable/
useful.   This has been my experience, in the extreme.

Secondly, people are looking for the commercial companies (note that
I am, and have been an employee of MSI for quite some time) to
do the benchmarks, and indeed, I can tell you that there has been for 
many years lively discussion within MSI regarding just that.  
But the first point keeps rearing its ugly head, and the more we
bring in 'consultants' and outside-academics, the more convoluted
things get.

My additional note is this: there is a 'Consumers Union' and 'Consumers
Report'
that does analysis and 'benchmarking' for a reason.
It is crucial, for them to be useful to the public at large, that such
tests be run in an independent, objective, repeatable fashion
(the hallmarks of good 'science', eh?)

I am *definitely* not suggesting that MSI is shy about having our 
products tested and compared; we do that evey day as we go
into companies world-wide.  Of course, evey customer before
they buy, wants to know how well you stack up against the competition.
I'm not here to soap-box for MSI;  I just don't want to give the
silly impression that I'm trying to 'hide' where I work.

But my main focus is, that given the recent fracas of linus 'benchmarks',
one might think that the populace at large might have an interest in
*establishing* what good benchmarks are, and how they are run
and by what criteria they are judged.  I think if that
could be done, MSI would be thrilled to compete.
But as with any 'reality check' the closer one looks, the more complicated
things become.   This is not to say that such efforts are not important, 
but that it isn't as simple as it implies to just say "someone should do it".

Don Gregory

Dr. Don Gregory ([email-masked])
Asia Pacific Science Liaison
Molecular Simulations Inc.
9685 Scranton Rd.
San Diego, CA  92121
(619) 799-5331     http://www.msi.com