Subject: Re: CCL:cross-validation and prediction with PLS

From: "Dr. Dave Winkler" <[email-masked]>
Date: Wed, 8 Sep 1999 09:35:25 +1000

>A requisite for the data is to be standardized (mean zero, sdv one).
To

>check the statistical significance of a predictive model the

>leave-one-out (LOO) cross-validation is usually performed. When a

>molecule is removed, a new standardization needs to be computed for
the

>remaining n-1 compounds, and the new mean and sdv will be applied to
the

>extracted molecule. However, for the usual sample sizes (n=30 to 150)

>the error committed in equalling the mean/sdv for the n-1 and n

>molecules is not neglectable, and it clearly affects the results of
the

>prediction for the removed molecule. The same can be said about the

>prediction of the property for a test set: the mean/sdv of the
training

>set QSAR model will be applied to the test set in order to have

>standardized descriptors, but the mean/sdv if we included all the

>molecules (training+test) may be very different.

There are a whole raft of problems which arise with cross-validation
and testing,  for example:

1.  What is the best way to chose a validation set (or test for that
matter) is leave-one-out, leave-N-out etc?

2.  Should clustering be used to choose a representative test set and ,
if so, what kind of clustering algorithm and do you cluster on X, Y or
X/Y?

3.  Effort involved in the cross validation process scales as the
square of the number of compounds and the square of the number of
independent variables, which has important implications for large data
sets (eg combichem/HTS data)

4.  Once a cross validation process has been completed, there are N
QSAR models, all slightly different.  Which is the 'best' or 'true'
model?

5.  What statitical methods should be used to choose the 'best' model?

6.  How do you avoid overfitting?

My colleague, Frank Burden (Monash University) and I have used Bayesian
regularized neural nets for QSAR.  We find that they overcome virtually
all of the problems with PLS QSAR models as they give the single
statistically best model possible for the data set.  In addition there
are good theoretical reasons why they do not require cross validation
or test sets.  We are investigating this for QSAR and preliminary
results suggest that this is the case.

We have published some of this work recently:

[74] New QSAR Methods Applied to Structure-Activity Mapping and
Combinatorial Chemistry, Burden, F.R. and Winkler, D.A<italic>. J.
Chem. Inf. Comput. Sci</italic>. <bold>39</bold>, 236 (1999).

[75] The Computer Simulation of High Throughput Screening of Bioactive
Molecules, F.R. Burden,  D.A. Winkler, <italic>in Molecular Modelling
and Prediction of Bioactivity</italic> (K. Gundertofte and F.S.
Jorgensen eds), Plenum Press 1998.

[80] Robust QSAR Models Using Bayesian Regularised Artificial Neural
Networks, Burden, F.R. and Winkler, D.A.  <italic>J. Med.
Chem</italic>., 1999; <bold>42</bold>(16); 3183-3187 (1999).

[81] A QSAR Model  for the Acute Toxicity of Substituted Benzenes
towards <italic>Tetrahymena Pyriformis</italic> using Bayesian
Regularized Neural Networks. F R. Burden*  David A. Winkler,
<italic>Chem. Res. Toxicol.,</italic> in press.

[82] Robust QSAR Models from Novel Descriptors and Bayesian Regularized
Neural Networks, Winkler, D.A, Burden, F.R.  <italic>Mol.
Simul</italic>. 1999 in press.

[87] Do QSAR Models using Bayesian Regularized Artificial Neural
Networks Really Need Validation?  Winkler, D.A. and Burden, F.R. 
<italic>J.Chem. Inf. Comput. Sci</italic> in preparation.

Cheers,

Dave

Dr. David A. Winkler                    Email:
[email-masked]   

Senior Principal Research Scientist     Voice: 61-3-9545-2477      

CSIRO Molecular Science			Fax:   61-3-9545-2446

Private Bag 10,Clayton South MDC 3169   http://www.csiro.au

Australia 	        		http://www.molsci.csiro.au