Subject: Re: CCL:Benchmarks

From: Bill Laidig <[email-masked]>
Date: Thu, 15 Apr 1999 08:06:28 -0400

David Feller's recent e-mail  concerning benchmarks and the difficulty of
head-to-head program comparison struck a chord with me.  As an industrial user I
am continually annoyed about lack of benchmarks, program inoperability between
vendors, lack of public validation suites, etc.  I agree with Dr. Feller that
until purchasers demand real benchmarks we won't get a very true comparison of
different program packages.  I feel the same is true for the other areas
I mentioned.  One way to cause these changes to happen would be for potential
purchasers to adopt a set of guidelines that, for example, by date 20XY  (a
reasonable implementation time period) these features/contract changes, etc. be
in the package.  If not, the packages would not be purchased if alternatives
exist.   Areas that I think are very basic and not adequately addressed by many
vendors are:

Benchmarking - some vendors have anticompetive clauses that restrict distributing
unfavorable benchmarks or do not provide adequate information to allow serious
benchmark comparison.  Such impediments to benchmarking need to be eliminated.

Molecule coordinate/definition interchange - many packages have very rudimentary
abilities to import/export molecule information accurately through common file
types.  Many only allow interchange by the "lowest common denominator" file
format such as PDB.  An agreement on one or more common interchange formats that
must be supported along with a test set to ensure compliance would go along way
to making components from different vendors interchangeable.  These formats
should be completely documented and nonproprietary.  This also would allow
conversion programs like Babel to concentrate on doing a really complete
implementation and lessen the possibility of information getting lost at each
translation step.

Standard output - In addition to any standard output format, for a given class of
programs such as electronic structure programs require a standard
results/archiving format so that results can be analyzed, entered into databases,
etc. by programs from multiple vendors.  Again this should be completely
documented and nonproprietary.

Standard input - This one is a little harder, but would be very useful.  In
addition to any standard input format, for a given class of programs, such as
electronic structure programs, require a standard input format (that allows
vendor extensions for their propretary features/methods, etc.) so that general
purpose programs could call different special purpose programs as long as the
same methods are implemented in the two specific programs.  An example would be a
QSAR module that needs to call an electronic structure code to get dipole
moments.  The user should be able to buy any code that delivers the appropriate
result and get the QSAR program to be able to call it.  Again this should be
completely documented and nonproprietary.

These are just a few example and there likely are several other areas that would
benefit from standardization/openness, etc.  I should mention that these are my
personal opinions and do not necessarily reflect those of my employer The Procter
& Gamble Company.

Thanks,
Bill

[email-masked] wrote:

> Over the past couple of days there has been a discussion about Jaguar,
> from Schrodinger, Inc.   Bill Glauser has called for comparisons with
> other codes, a call which I would second.  In the past, PNNL has supported
> a benchmark activity, which led to a report that was made available over the
> web.  Unfortunately, constructing such a meaningful benchmark is a
> considerable amount of work and not everyone is as happy as Schrodinger, Inc.
> to have their software compared with the competition.
>
> In my experience, each of the major electronic structure packages has it's
> strengths and weaknesses.  In part, that's why it's a nontrivial exercise to
> run a useful set of benchmarks.  It's much easier to simply take any set
> of calculations that are handy and quickly run them.  Such results are only
> one step removed from purely anecdotal reports of how much faster code A
> is than code B.
>
> Until purchasers of these packages support (demand?) the sorts of head-to-head
> comparisons that you might find in a Consumer's Report article on family
> sedans, people will have to be content with information that is far from
> complete.
>
> Dave
>
> --
>
> David Feller                                | Mail Stop K1-96
> Environmental Molecular Sciences Laboratory | Box 999
> Battelle Pacific Northwest National Lab     | Richland, WA 99352
>                                             |
> e-mail:[email-masked]                  | Fax: (509)-375-6631
>
> ---
> Administrivia: This message is automatically appended by the mail exploder:
> [email-masked]: Everybody | [email-masked]: Coordinator
> [email-masked]: HELP CHEMISTRY or HELP SEARCH | Gopher: www.ccl.net 73
> Anon. ftp: www.ccl.net   | [email-masked] -- archive search
>              Web: http://www.ccl.net/chemistry.html
> ---

--
************************************************************************
*    "Like jewels in a crown, the precious stones glittered in the     *
*     queen's round metal hat." - Jack Handey                          *
*                                                                      *
*     Bill Laidig                                                      *
*     The Procter & Gamble Co.             tel 513-627-2857 fax - 1233 *
*     Miami Valley Laboratories            [email-masked] (preferred)   *
*     P.O. Box 538707                      [email-masked]            *
*     Cincinnati, OH 45253-8707                                        *
************************************************************************