Subject: RE: CASP vs. The Tower of Babel

From: "LOEFFLER,DR,GERALD FEM BENAT" <[email-masked]>
Date: Wed, 23 Sep 1998 09:39:58 +0200

Hi!

So why is there no collaboration on the code-level between computational
scientists?

I don't want to over-simplify the problem, but one important reason IMHO
is the fact that traditionally computational scientists have been
_lousy_ programmers. While the technology to make interoperable
application components has been there for some time (I'm particularly
thinking of OO technology), computational scientists have been too
inexperienced in software engineering techniques to exploit this
technology.

As a concrete example: It would be almost trivial to design an OO
framework for Energy Minimization (or Molecular Dynamics or Threading,
 ...) where the force-field would be just one component that you "plug"
into the framework. The same is true for such algorithmic components as
minimization strategies, potentials of mean force, integration
algorithms, Monte Carlo moves, Genetic Algorithms, etc.

But is the resulting code efficient? It's certainly a lot slower than
FORTRAN! (For language reasons and because you loose opportunities for
source-level optimizations if components have to be truly independent of
each other.) But I would argue that firstly speed of execution does not
matter at this level (we are not talking about production code here).
And secondly, on the other hand, of what use is speed of execution if
you can not get the functionality you want (interoperability of
algorithms, in this case) with reasonable effort?

But things _are_ getting better: MMTK springs to my mind, and also the
OMG effort to standardize interfaces for biological sequence analysis
(although this is not an academic effort). And of course there are
programs which are not designed with OO techniques and consequent
interoperability in mind, but which are simply well written (TINKER, for
instance).

(Regarding unwillingness to give away source code at all: With Java you
can even give away the "object code" (byte code) (maybe after running it
through an obfuscator) and it will work on any Java platform...)

	cheers,
	gerald

|--------------------------------------------------------------------|
 Gerald Loeffler - Bioinformatics Scientist
 Boehringer Ingelheim R&D Vienna
 Email: [email-masked]
 Phone: +43 676 3289588 (and +43 1 80105 634)
 Fax:   +43 1 80105 683
 Smail: Bender+Co, Dr. Boehringer-Gasse 5-11, A-1121 Vienna, Austria

> -----Original Message-----
> From:	Gabriel Berriz [SMTP:[email-masked]]
> Sent:	Sunday, September 20, 1998 7:53 PM
> To:	[email-masked]
> Subject:	CCL:CASP vs. The Tower of Babel
> 
> 
> 
> I study the statistical mechanics of protein folding using minimal
> computer models.  I find that my specialty has a Tower of Babel
> (Babble?) problem, and perhaps the same is true of the whole field of
> computational protein studies, or even of all of computational
> chemistry.  It has to do with checking and building on the results of
> others.  I was an experimentalist in cellular immunology for a few
> years before switching to my current field, and I recall that trading
> reagents, libraries, strains, was quite common in that field.  I often
> unpacked little vials shipped in dry ice, and bearing some precious
> mutant; typically, after some quick tests, I was up and running with
> the new stuff.  Not much was required from the source of the samples
> (a couple of concentrations, buffers used, maybe some growth
> conditions here and there...).
> 
> *Nothing* like this happens in my current field.  I'm not sure why,
> but I have a few guesses.  For one thing, programs are a pain to port
> across systems (portability of code is not a criterion for
> publication).  More important, in most cases I don't want the program
> just to use as a black box.  On the contrary, my interest in the
> program is usually in how it implements a model; I want not only to
> reproduce published results, but also to tweak the conditions, and to
> extend the experiments.  This invariably requires that I understand
> enough of the code to hack away in it, and here's where I hit the
> biggest wall.  It takes me too long to understand the code written by
> my *labmates*, let alone that written by some unknown graduate student
> 5 years ago half a world away.  (Again, clarity of code is, for the
> most part, not a criterion for publication).  So, typically I conclude
> that either I re-implement the idea from scratch, which is usually
> something I can't afford, or else I drop the matter altogether.
> (Incidentally, in the few cases I've tried to get source code from
> other labs, I've received such unequivocal, resounding, unapologetic
> refusals, that I must conclude my request was deemed to be bad
> manners.)
> 
> It is, in my opinion, a very serious problem; it reduces the field to
> a collection of largely independent efforts, deprived of one of the
> greatest strengths of the scientific method, namely, the ability to
> test and build upon the work of others.  I wonder if others feel
> similarly.
> 
> I think this frustrating situation was what ultimately gave rise to
> the biennial structure prediction competition CASP (Critical
> Assessment of techniques for protein Structure Prediction), in which
> participants put their structure prediction programs through the fire
> test of predicting some recently solved protein structures prior to
> their publication.  This skips over the problem of understanding the
> programs and the models devised by others by focusing on "objective
> results".  Faced with this clear prize, the field has naturally
> responded by a adopting an increasingly heuristic attitude: whatever
> works, however ad hoc or poorly understood, throw it in there.  If you
> loose, no one will care, and if you hit the CASP jackpot, then
> "there's no arguing with success!"
> 
> Well, I guess that's *one* way to deal with our Tower of Babel
> problem, but I wonder where this leaves the science...  I'm relatively
> new to this field, though, and I wonder what others with more
> experience feel about these issues.
> 
> Best wishes,
> 
> Gabriel Berriz
> Department of Chemistry and Chemical Biology
> Harvard University
> [email-masked]
> For best results, replace the word potato by chasma in my address.
> 
> 
> ---
> Administrivia: This message is automatically appended by the mail
> exploder:
> [email-masked]: Everybody | [email-masked]:
> Coordinator
> [email-masked]: HELP CHEMISTRY or HELP SEARCH | Gopher:
> www.ccl.net 73
> Anon. ftp: www.ccl.net   | [email-masked] -- archive
> search
>              Web: http://www.ccl.net/chemistry.html 
> ---

---

From: Konrad Hinsen <[email-masked]>
Date: Fri, 25 Sep 1998 10:36:24 +0200

> into the framework. The same is true for such algorithmic components as

Of course; MMTK does it that way, and it works just fine.

> each other.) But I would argue that firstly speed of execution does not

Loss of optimization is not a problem; if you consider what takes most
of the time in computationally intensive algorithm, it's always energy
evaluation, and that's always in one module (and a rather small one).
The one point where I had to compromise for efficiency reasons is the
interface between force field evaluation and the definition of the
simulation universe. The latter defines how distances between atoms
are calculated (e.g. minimum image convention), so the code should be
in the universe module and not in the force field module. But distance
calculations are used so frequently that the performance penalty of an
indirect function per distance evaluation would be too high. I settled
for a scheme that hard-codes the distance calculation into the force
field routines for the common cases and uses an indirect function call
for other (to be defined) univers shapes.

Language/compiler issues are a different story. Since the numerics
community has never shown much interest for anything but Fortran until
recently, compiler manufacturers were under no pressure to work on
code optimization for numerical problems in other languages. Some
compilers do better than others; with a simple test case (hand-coded
matrix multiplication) I found C/Fortran speed ratios between 1.1 and
100, depending on the machine. Moral: check compiler performance
before buying a machine.

However, you can still use Fortran for time-critical code (which is
always a small part of a simulation program) and another language for
the rest, at the price of worrying about machine-dependent
interlanguage linking. I regularly use Fortran subroutines with
MMTK, although the public distribution contains f2c-compiled
code (less efficient) because I don't want to deal with questions
about Fortran-C linking.

In my opinion, the most important shortcoming in Fortran 9x is the
lack of a standard interface with C. Such a standard interface
would have made mixed-language programming the best solution for
scientific software development (even if you don't like C, almost
all languages have interfaces to C, so even Fortran-Pascal or
Fortran-Lisp would be possible).

> through an obfuscator) and it will work on any Java platform...)

True, but it's still like binary code; people can't inspect and modify
it. And I would never use such programs for my scientific work, as far
as calculations are concerned (I don't care about the source code for
my text editor, of course).
-- 
-------------------------------------------------------------------------------
Konrad Hinsen                            | E-Mail: [email-masked]
Centre de Biophysique Moleculaire (CNRS) | Tel.: +33-2.38.25.55.69
Rue Charles Sadron                       | Fax:  +33-2.38.63.15.17
45071 Orleans Cedex 2                    | Deutsch/Esperanto/English/
France                                   | Nederlands/Francais
-------------------------------------------------------------------------------

---

From: Rick Venable <[email-masked]>
Date: Fri, 25 Sep 1998 12:03:31 -0400 (EDT)

> So why is there no collaboration on the code-level between computational

There is; NETLIB is a good example, and there are a fair number of other
ftp and web sites with excellent code available, covering a variety of
scientific computational disciplines.  The problem comes when you get to
comprehensive packages in more specific application areas, such as those
associated with computational chemistry.

For that case, the answer is often money.  For one thing, many of the
better packages with academic roots have "gone commercial" in one way or
another.  Also, some academic groups use a modest charge for their
source code as a means to fund the coding effort that supports their
research.  In some cases, universitities have chosen to "market" the
products of their research groups, including research computer programs. 
In all of these cases, redistribution of source code is prohibited,
although it may be available to non-profit researchers at little or no
charge.

An underlying issue is the ease-of-use factor; it still takes
considerable effort to design and implement a rigorous and friendly
graphical user interface, and someone needs to be paid to do that.
Documentation is another factor, as is long term technical support
(bugfixes, new features, etc.).

--
Rick Venable                  =====\     |=|    "Eschew Obfuscation"
FDA/CBER Biophysics Lab       |____/     |=|
Bethesda, MD  U.S.A.          |   \    / |=|  ( Not an official statement or
[email-masked]  |    \  /  |=|    position of the FDA; for that,
http://www.erols.com/rvenable       \/   |=|    see   http://www.fda.gov  )