Subject: CASP vs. The Tower of Babel

From: Gabriel Berriz <[email-masked]>
Date: Sun, 20 Sep 1998 13:52:43 -0400

I study the statistical mechanics of protein folding using minimal
computer models.  I find that my specialty has a Tower of Babel
(Babble?) problem, and perhaps the same is true of the whole field of
computational protein studies, or even of all of computational
chemistry.  It has to do with checking and building on the results of
others.  I was an experimentalist in cellular immunology for a few
years before switching to my current field, and I recall that trading
reagents, libraries, strains, was quite common in that field.  I often
unpacked little vials shipped in dry ice, and bearing some precious
mutant; typically, after some quick tests, I was up and running with
the new stuff.  Not much was required from the source of the samples
(a couple of concentrations, buffers used, maybe some growth
conditions here and there...).

*Nothing* like this happens in my current field.  I'm not sure why,
but I have a few guesses.  For one thing, programs are a pain to port
across systems (portability of code is not a criterion for
publication).  More important, in most cases I don't want the program
just to use as a black box.  On the contrary, my interest in the
program is usually in how it implements a model; I want not only to
reproduce published results, but also to tweak the conditions, and to
extend the experiments.  This invariably requires that I understand
enough of the code to hack away in it, and here's where I hit the
biggest wall.  It takes me too long to understand the code written by
my *labmates*, let alone that written by some unknown graduate student
5 years ago half a world away.  (Again, clarity of code is, for the
most part, not a criterion for publication).  So, typically I conclude
that either I re-implement the idea from scratch, which is usually
something I can't afford, or else I drop the matter altogether.
(Incidentally, in the few cases I've tried to get source code from
other labs, I've received such unequivocal, resounding, unapologetic
refusals, that I must conclude my request was deemed to be bad
manners.)

It is, in my opinion, a very serious problem; it reduces the field to
a collection of largely independent efforts, deprived of one of the
greatest strengths of the scientific method, namely, the ability to
test and build upon the work of others.  I wonder if others feel
similarly.

I think this frustrating situation was what ultimately gave rise to
the biennial structure prediction competition CASP (Critical
Assessment of techniques for protein Structure Prediction), in which
participants put their structure prediction programs through the fire
test of predicting some recently solved protein structures prior to
their publication.  This skips over the problem of understanding the
programs and the models devised by others by focusing on "objective
results".  Faced with this clear prize, the field has naturally
responded by a adopting an increasingly heuristic attitude: whatever
works, however ad hoc or poorly understood, throw it in there.  If you
loose, no one will care, and if you hit the CASP jackpot, then
"there's no arguing with success!"

Well, I guess that's *one* way to deal with our Tower of Babel
problem, but I wonder where this leaves the science...  I'm relatively
new to this field, though, and I wonder what others with more
experience feel about these issues.

Best wishes,

Gabriel Berriz
Department of Chemistry and Chemical Biology
Harvard University
[email-masked]
For best results, replace the word potato by chasma in my address.