Subject: Re: CCL:software evaluation / validation

From: Jacco van de Streek <streek~at~ccdc.cam.ac.uk>
Date: Sat, 06 Mar 2004 18:38:44 +0000

Andras.Borosy~at~givaudan.com wrote:
> I agree with the first part, which is indeed a _VERY_ important point:
> "The main criterion of scientific work is reproducibility."
> However, my question about this would be:
> How is it possible  to reproduce scientific results obtained by commercial
> _OR_ academic software which relies on _RANDOM_ methods ?

Hi,

I happen to work on a program that uses random numbers (to solve crystal
structures from X-ray powder data) and I think that reproducibility is not an
issue, at least not in the sense you are referring to.

Because it is a random process, users of such programs would not run them just
once, as the single outcome of a random event is meaningless. Instead, our
program by default runs the simulated annealing run ten times (with different
seeds for the random number generators). It is now the reproducibility (the
statistical distribution of solutions) itself that should be reproducible.

The fact that two different runs give two different results can actually be used
to validate the random method and its solutions (within the limitations of the
program, input data and input model). For example, if the ten runs give 8
identical solutions, with a figure of merit that is better than the other two
runs, one can be fairly confident that the "global" minimum was found (within
the limitations of the program, input data and input model). The two other runs
are also interesting: are they two wildly different solutions with similar
figures of merit that are significantly worse than the 8 better solutions? If
the other two solutions are identical and have good figures of merit, is there a
strong similarity between them and the eight better solutions?

If less than five similar solutions were obtained (in ten runs), with
siginificantly better figures of merit, this points to a problem with the search
space, and more exhaustive runs should be run. If several wildly different
solutions are found, with similar figures of merit, either there is a problem
with the search space (the global minimum has not been found yet), or the model
does not fit the data unambiguously (essentially an error in the input, be it
the data or the model).

Of course, in the publication all runs and their results (even if only in
general terms) should be mentioned.

Best wishes,
-- 
Jacco van de Streek
Research Scientist
Cambridge Crystallographic Data Centre
Cambridge, United Kingdom

---

From: "Dr. Csaba Hetenyi" <csaba~at~ovrisc.mdche.u-szeged.hu>
Date: Sat, 6 Mar 2004 14:54:22 +0100 (CET)

Hi,

> Source code is not necessary for testing or validation. It is only 
> necessary  if you want to modify, enhance, integrate, build onto an existing code 
> base. 

I mean, it is necessary for correct documentation. Obviously, a paper
cannot include all (computational) details of a new program or
methodology.
The source code could serve as an "appendix" in this case.
The users should have the right to check the details (source=appendix).

> (e.g. GOLD, AutoDock, ICM-dock etc. employ random methods).

Many of these methods are based on global search strategies including
Monte Carlo, Genetic algorithm, etc.
As these algorithms include random number generations, the test of these
methods could be done statistically, i.e. you make several trials and
check if you got acceptable results or not. (Statistical test is the ONLY
possibility to check the reliability of such randomized methods.)
---
As docking is a relatively "new" tool among other computational methods,
there is a lot to be done in testing and improving the technique and also 
the search strategies.
Despite its novelty and many simplifications included, this technique
provide surprisingly good results in many cases. 
In my view, computational (modelling) work, like docking should be
reproducible, otherwise we cannot expect experimental scientists to
collaborate us and trust our results.
In case of docking, e.g. the statistical reproducibility of structural fit
(RMSD) is a precise test of the method.
Best wishes,
Csaba