Subject: Benchmarks and software

From: David Giesen <[email-masked]>
Date: Mon, 19 Apr 1999 17:11:03 -0400

I could not agree more with the idea that all software should be subject
to a 'standard' benchmarking.  A polite (since this list is moderated)
pox on those vendors out there who make 'non-competition' a precondition
to use of their software.

I would like to add a second requirement of all commercial software if I
may dream a little.  All error messages produced from within the code
(i.e. non-system errors) should be written in plain user language, not
programmer debugging language.  Soon the day will come when I refuse to
spend $xx,xxx US$ to upgrade to a code which simply returns 'Error in
subroutine xxx' or even more simply, 'y=a' (I would like to give more
specific examples, but wouldn't you know, my license agreements prevent
me from doing that).  Another semi-polite pox on those guilty vendors.

At risk of touching on a controversy which almost flared up several days
ago, I'd like to point out that Jaguar does a nice job of printing
useful error messages and has very responsive help staff.

Dave

I am not affiliated with Schrodinger, Inc. (Jaguar), nor are my comments
reflective of the opinion of the Eastman Kodak Company.

-- 
Dr. David J. Giesen
Eastman Kodak Company                           [email-masked]
2/83/RL MC 02216                                (ph) 1-716-58(8-0480)
Rochester, NY 14650                             (fax)1-716-722-2327

---

From: Steven Bachrach <[email-masked]>
Date: Wed, 21 Apr 1999 08:30:26 -0500

> to use of their software.

Dave makes a good point here but I think a strong word of caution is
needed here. What would constitute a "standard" becnhmark for ab initio
codes? When I installed G98 I did not run all of the test jobs provided
since many of them cover types of calculations that I never run. What
might be an important calculation for me might be of little interest to
you.

Furthermore, benchmarks of ab intio codes are very much dependent on the
hardware and the compilers used. A particular code might have been
tightly tuned for a specific machine and perform quite well, and then
placed on a different machine the performance can degrade. This might
even be found with different memory/disk configurations.

Dave Feller has commented on many of these issues in the past - his
becnhmark report is quite revealing and interesting in just how
difficult it is to really 'benchmark' this type of program.

Benchmarking MM codes is probably an even bigger poroblem - not only are
you interested in speed, but the quality of the calculation is also in
question. (Obviously all ab intio codes should provide the same energy
for a HF/6-31G* calculation of methane, right?)

I would imagine that vendors should be willing to provide the codes for
local testing for a short period before making a purchase. If they
don't, you should demand it or walk away - would you buy a car without a
test drive?

Steve

-- 
Steven Bachrach				
Department of Chemistry and Biochemistry
Northern Illinois University
DeKalb, IL 60115                          Phone: (815)753-6863
[email-masked]                      Fax:   (815)753-4802

---

From: Peter Shenkin <[email-masked]>
Date: Wed, 21 Apr 1999 12:53:45 -0400 (EDT)

Warning:  long, rambling and philosophical response follows.

On Mon, 19 Apr 1999, David Giesen wrote:
> may dream a little.  

Where there's life, there's hope. :-)

> All error messages produced from within the code

Hi, Dave,

Though what you suggest is useful to strive for, I have found that even
when one makes great efforts to do this, users often don't understand
or believe the messages anyway.  Ideally, one would like to tell the 
user "you did this wrong" (when s/he did :-) ), but a given exception 
deep in the code can often be caused by a variety of problems with
the program input.

After many years of trying to deal with this as a code developer, I've
concluded that the most I can really hope for most of the time is to
write error messages that are going to make enough sense to me that if
a user reports them, it will be obvious enough to me what the problem
is to allow me to respond quickly.  Even this is terribly difficult.
The messages you quote would not pass even this test, however.

On the other hand, I have found that some users, when they see an
error message, simply won't believe what they read.  In MacroModel/BatchMin,
if a forcefield does not have parameters for a particular system, we
print out a line that says that parameters are missing for the following
(stretches, bends, or whatever).  The atoms in question are fully
identified.

One would think this is pretty clear.  However, I get at least 20
queries a year from people asking me what this means.  I always
respond by saying "It means that the force field does not have
parameters for certain stretches, bends, etc. in your molecule."
Somehow, they understand this perfectly when I say it in email,
even though the error message said exactly the same thing.  Go figure. :-) 

Of course, there may be 200 users who *do* understand, and who don't
have to call me, to the 20 who don't understand.  At least I hope so. :-)

Waxing philosophical for a moment, there are several mental models
involved.  One is the model of how any molecular mechanics program
works:  there are force-fields, interactions, energy terms;  then
there are minimization, conformational search, dynamics, etc.  Not
all users of these programs have this model in their heads.  They
may not have a good idea of what an interaction is, or even what a
force-field is.  Thus, my explanation may not even be in the language
that the (naive) user understands.  S/he just knows that "something
failed."

Another mental model involves the way a particular program does
things -- how it chooses to carry out, say, conformational search.
Even fewer users of a program will have this model in their heads.
In fact, only the most experienced users will have it, and even
then they will only have a partial picture.  More and more, the
same is true even of developers, especially for legacy codes,
but I digress.  

For example, if a conformational search aborts with a message 
like, "You are trying to alter 10 degrees of freedom but only 9 
were specified", that is very, very descriptive, but only if you 
have the right model in your head.  In our program, it means
that you requested that up to at 10 torsions or intermolecular 
translations/rotations be varied in a single step, but you
only specified 9 variable torsions or intermolecular motions
that can be varied at all.

Only a sophisticated user of our particular program would be able
to understand this.  But at least I can understand it immediately
and respond to the user giving a variety of possible causes.

	-P.

--
********* Peter S. Shenkin; Schrodinger, Inc.; (201)433-2014 x111 *********
*********** [email-masked]; http://www.schrodinger.com ***********