Subject: CCL: cross-validation and prediction with PLS

From: David Robert <[email-masked]>
Date: Tue, 07 Sep 1999 13:53:37 +0200

Dear CCLers,

A question on cross-validation and prediction in QSAR models using PLS:

A requisite for the data is to be standardized (mean zero, sdv one). To
check the statistical significance of a predictive model the
leave-one-out (LOO) cross-validation is usually performed. When a
molecule is removed, a new standardization needs to be computed for the
remaining n-1 compounds, and the new mean and sdv will be applied to the
extracted molecule. However, for the usual sample sizes (n=30 to 150)
the error committed in equalling the mean/sdv for the n-1 and n
molecules is not neglectable, and it clearly affects the results of the
prediction for the removed molecule. The same can be said about the
prediction of the property for a test set: the mean/sdv of the training
set QSAR model will be applied to the test set in order to have
standardized descriptors, but the mean/sdv if we included all the
molecules (training+test) may be very different.

I am afraid that LOO cross-validation is sometimes calculated simply by
recomputing the regression coefficients, keeping fixed the scores of the
n molecules model, which obviously overrates the results. Any opinions?

David Robert

--------------------------
E-Mail: [email-masked]
Institute of Computational Chemistry
University of Girona
Campus Montilivi
17071 Girona, Catalonia, Spain

---

From: [email-masked]
Date: Tue, 7 Sep 1999 14:39:55 +0100

David

Basically, you are asking the right questions about PLS/CV.
There are a number of publications that discuss these issues; e.g.,
Cramer RD in 3D-QSAR in Drug Design, 1993 and also in Perspect.
in Drug Disc. and Design, 1 (1993), 269-278.

First, it is not a pre-requisite that the data be standardised in the way
you describe (Z-score autoscaling). In fact the so-called zero th PLS 
LV corresponds to mean-centring the data. Adjusting the univariate
stdev to one is optional and makes sense with some data and
not others. Univariate autoscaling does not make sense, for
example, with CoMFA where its application means that high
energy interactions have equal prior weighting as low energy
interactions -  the former _may_ have very high variance while
the latter cannot have high variance. PLS is sensitive to both
the univariate X-block variance and the strength of the univariate
correlations with Y. Autoscaling can make sense where for
example descriptors from different sources are used.

With regard to data (re)scaling during crossvalidation it appears
that different groups favour different approaches. The Tripos (ie 
CoMFA etc) and GOLPE methods both rescale the "kept-in" rows 
(compounds) for each CV cycle. I personally favour this approach.
The left-out compounds are rescaled/centred according to the stats 
for the kept-in rows. This means that the predictions are "true" 
predictions; i.e., if the means/stdevs used are not recalculated for each 
CV cycle then information about the left-out compounds has 
been included in the model development stage and are thus not true 
predictions. The expectation is that the latter method will produce more 
optimistic CV stats (ie q2) than the former approach. The latter method 
is in fact used by the Umetri software (SIMCA, MMODE etc) although 
I may be out-of-date here. It is also an inconsistent approach in the sense
that external test set predictions are made without including information
about these compounds in the model development stage.

The only argument I know of in favour of the Umetri CV approach is that the 
omission of a single row may dramatically change the univariate characteristics
of one or more X columns and thus lead to "strange results" (Thibaut et al,
QSAR Journal, 13, 1-3 (1994). Presumably, these authors mean that an
inappropriate number of PLS LVs may be chosen as optimal and, therefore,
external test set predictions may be poor. It is probably appropriate to
analyse ones data for such columns and determine the influence such
columns have on the final model (ie omit that/those column/s and compare q2 
and optimal LV nos. etc). I don't know of any literature on this however.

The most significant influence on PLS CV results is generally the level of 
row/compound redundancy. Sparse, designed training sets will tend to
produce poor q2 scores (but may have very good external predicitvity) while 
highly redundant datasets will tend to give high q2s particularly (but not solely) 
where LOO CV rather than LNO CV is used. So it can be useful to do some
experimental design and then add some redundancy (!). So, the "real" test
of ones model must always be performance with external test compounds.
Also, look at the effects of choosing one more or one less LV on your
predictions, together with a search for leverage (outliers) in both training
and test sets. 

Hope this help and sorry if this is a little unstructured and rambling ...

Dave

> n molecules model, which obviously overrates the results. Any opinions?

Dr David Turner
Dept of Information Studies, Sheffield University
Sheffield, S10 2TN, UK        Tel. 0114 2 222 650
E-mail: [email-masked]
Fax: 0114 2 780 300

---

From: David Larson <[email-masked]>
Date: Tue, 07 Sep 1999 10:26:25 -0500

> n molecules model, which obviously overrates the results. Any opinions?

One does indeed need to be careful in using PLS packages to find out
whether *full* cross-validation is done, and it becomes a particularly
relevant point when trying to compare published statistics with what you
obtain yourself.  The PLS cross-validation procedure used for CoMFA in
SYBYL QSAR has been described in detail[1].  There, as should always be
the case, each cross-validation PLS is calculated "from scratch" in that
the reduced data matrix is re-centered and rescaled for each
cross-validation run (rescaling does not affect CoMFA much, however,
because each molecular field is scaled as a block).  This almost always
produces a more conservative (and less biased) *estimate* of q^2 [2]. 
Note that the actual predictivity of the model [3] is a *population*
parameter and is independent of how the sample statistic q^2 is
calculated.

My own belief is that the standard error of prediction is what should
always be the primary criterion used for comparing models, whether it is
>from cross-validation or for external test sets.  Predictive correlation
coefficients (q^2's) are dangerous because they can be very dependent on
the distribution of the training set used.  Pre-selecting extrema as a
training set and using more centrally distributed compounds as the test
set, for example, will usually "improve" q^2 even if the error of
prediction is actually higher.  This is because the training set SD
increases even more that the SE of prediction.  This is similar to the
reason that the SD of the test set should not be used in calculating
predictivity. 

A related, more subtle concern arises up when variable selection is
done.  There, observations should properly be left out from the very
start of the process and multiple variable selections run and compared. 
Unfortunately, cross-validation is sometimes only run after the variable
set has already been selected.

Bob Clark  

*******************************************************************************   

[1] RD Cramer III, SA DePriest, DE Patterson & P Hecht: The Developing
Practice of Comparative Molecular Field Analysis. In "3D QSAR in Drug
Design: Theory, Methods and Applications," H. Kubinyi ed.; ESCOM,
Leiden, 1993; p 461.

[2] H Kubinyi & U Abraham: Practical Problems in PLS Analyses.  In "3D
QSAR in Drug Design: Theory, Methods and Applications," H. Kubinyi ed.;
ESCOM, Leiden, 1993; p 722.

[3] Would that be gamma^2, or perhaps xi^2? I have never seen a name for
this, in the way that rho^2 corresponds to r^2.

**************************************************************************
**************************************************************************
**                                                                      **
**    Dr. Robert D. Clark                     [email-masked]         **
**    Senior Development Scientist            phone: 314-647-1099       **
**    Tripos Inc.                               fax: 314-647-9241       **
**    1699 S. Hanley Road             _____                             **
**    St. Louis MO 63144             /     \                            **
**                                  ||      |                           **
**                                 C   O \ O |                          **
**                                  ||  (_> |                           **
**                                  \ \/--\/)                           **
**                                   \_____/                            **
**                                                                      **
**************************************************************************
**************************************************************************