Subject: Re: CCL:Enhancing CoMFA

From: "David B.Turner" <[email-masked]>
Date: Fri, 18 Dec 1998 11:29:50 +0000

Dear all

To add to the previous message from Bob Clark:

I agree with everything he has to say about CoMFA grid
resolution / sampling-related effects. I didn't realise that Arthur 
Wang was suggesting that you take the orientation with the highest 
q2; this is clearly the wrong thing to do.

I have myself performed some extensive analyses of the effects 
on q2, r2 and test predictive r2 (pr2) of aggregate-body reorientation 
within the 3D-grid. I have done this at both 1 Ang and 2 Ang for 
two datasets.

The results, from >1,000 different orientations, are (in brief):

Melatonin receptor ligands (Sicsic et al, JMC, 1997, 40, 739)
            1 Ang                                                 2 Ang
q2   0.67 - 0.76    Mean 0.72          0.58 - 0.81  Mean 0.69 
r2    0.85 - 0.97    Mean 0.93          0.72 - 0.97 Mean 0.87 
pr2  0.66 - 0.80   Mean 0.73           0.43 - 0.88  Mean 0.70

Steroids (from original CoMFA paper but corrected by Wagener et al,
JAmChemSoc, 1995, 117. 7769 ):
(Fluorine-containing compound omitted from prediction set)
            1 Ang                                            2 Ang
q2    0.85 - 0.88   Mean 0.86            0.76 - 0.90   Mean 0.85 
r2    0.93 - 0.94   Mean 0.93             0.88 - 0.96  Mean 0.93 
pr2  0.81 - 0.86    Mean 0.84            0.63 - 0.91  Mean 0.83

In all cases there is almost zero-correlation (r) between q2 and
test set pr2 which clearly demonstrates the problem with taking
the orientation with the maximum q2 (although this doesn't really
need demonstrating...)

The steroid results are pretty stable (low variance) at 1 Ang while the 
melatonin scores are less so. I haven't looked at the variation in prediction for
 individual compounds - I have the data however. The 2 Ang results
(see also Cho and Tropsha CV-R2 GRS) suggest that many published 
PLS results are pretty meaningless**. My results for test set pr2 show
this even more strongly than with q2. Overall, the probability of
getting a model significantly different from the mean ("true"?) model is
quite low at 1 Ang but you _don't know_ where in the distribution your
initial model is unless you do some kind of reorientation testing ...
I think this will be the case whether or not Sybyl "ORIENT BEST_VIEW"
is used.
 
The steroid results will appear in a paper about EVA (in press 
JCAMD, Trevor Heritage and Allan Ferguson are co-athors). The
melatonin results have been submitted as part of a paper
about the application of a GA to selecting localised
Gaussian kernel widths in an EVA analysis.

Surely, the most straightforward means
of improving model-reliability is to use the smallest grid-spacing
possible / reasonable with the resources available. The use of 
SAMPLS makes this feasible for 1 Ang. resolution and less 
(depending on the number and size of compounds, of course). Once the 
q2 resulting from a particular resolution approximates to that of 
the mean/mode/median q2 from many analyses at, say, 2A 
resolution then one can say that you may have a reliable model 
that can be used for test set prediction.

On a slightly different subject why doesn't Tripos make general (rather 
than  HQSAR-only) use of  SAMPLS to perform fitted analyses and 
test predictions? It's very easy to implement.

Regards

David Turner

** This is ignoring: dataset design aspects which  is another kettle 
of fish, the type of CV to use and its relationship to dataset
design or lack of etc ... Experimental errors for bioactivity
data are also apparently generally ignored so the precision of
some estimates may be unwarranted ... 
Dr David B. Turner
Dept. of Information Studies
University of Sheffield,
Sheffield S10 2TN
UK
Tel.: 0114 2222 650   Fax.: 0114 2780 300
E-mail: [email-masked]