Subject: QSAR and scarmbling

From: "O.Nicolotti" <[email-masked]>
Date: Fri, 19 Oct 2001 15:09:54 +0100

Hy everybody,
comments on scrambling QSAR data have been reported as follows

thanks,
Orazio 

Hy James
and thanks a lot for your help.
I have been reading your paper you referenced. 
Basically your results summarised in table 5 seems to confirm my 
initial feeling that r2 values are not relevant after scrambling data. 
And in fact your results (linear equation, but the same can be 
extended to the spline one) coming from G/PLS approach as on 
MFA as on Descriptor Data showed that the r2 values are still quite 
high (0.714 and 0.709 respectively) while r2(CV) values drop down 
(0.129 and 0.266 compared to 0.789 and 0.728 of the 
corresponding unscrambled data).
According to your results, I would say that scrambling is effective 
only to validate predictive QSAR power which means to look only 
at r2(CV) values before and after scrambling. Therefore r2, which 
accounts only for the model fitting is not worth to be investigated in 
scrambling.

If you agree this conclusion I will make a recap for any CCL users. 
Please let me know

Still thanks for your mail.

Orazio 

> 
> Hi Orazio
> 
> Theres a couple of papers that tackle part of this, they are
> referenced in a paper I did on organophosphates (Drew, M.G.B., 
Lumley,
> J.A. and Price, N.R., Predicting ecotoxicology or 
organophosphorous
> insecticies: Successful parameter selection with the genetic 
function
> algorithm., Quant.Struct.Activity Rel. 1999 18 573-583).  We 
found
> scrambling a useful technique for validating regression equations 
when
> using approaches like GAs to select the best descriptors (e.g.: 
> from
> CoMFA like analysis).  If I remember correctly the original papers 
(by
> Toplis and Edwards ?) looked at the effect of increasing the 
number of
> original variables used to select the regression equation 
components. 
> They found that the r2 increased with the number of original 
starting
> variables regardless of the number of descriptors in the final
> equation.  Thus r2 alone does not indicate whether you have 
found a
> chance regression equation.  We found that scrambling the 
activity
> data does give you some indication and r2(CV)s helped.  There 
may be
> some other papers we referenced of interest.
> 
> hope this is of some help,
> 
> james
> 
> ************************
> Dr. James A. Lumley
> Arrow Therapeutics
> Woodmansterne Road
> Carshalton Surrey
> SM5 4DS
> Tel: 0208 652 9618
> Fax: 0208 652 9620
> http://www.arrowt.co.uk
> ************************
> 
> Confidentiality Notice:
> The contents of this email from Arrow Therapeutics are 
confidential to
> the ordinary user of the email address to whom it was addressed.
> No-one else may copy or forward all or any of it in any form. If you
> receive this email in error please reply to the sender so that we 
can
> arrange for proper delivery and then please delete the message 
> from
> your inbox.   Thank You.
> 
> 
> -----Original Message-----
> From: Computational Chemistry List
> [mailto:[email-masked]]On Behalf Of O.Nicolotti Sent:
> Saturday, October 13, 2001 3:09 PM To: [email-masked] 
Subject:
> CCL:QSAR and scrambling
> 
> 
> Hy everybody,
> I would like to know your opinion about scrambling the activities 
data
> for validating QSAR models.
> 
> Usually, after scrambling affinity data, the classical statistics
> parameters (r2 and q2) are supposed to go dramatically down if 
the
> original QSAR models was a reliable model.
> 
> Anyway, if you are dealing with CoMFA, I would expect bad q2
> after scrambling but the r2 could also be still quite high since r2
> tend to be high when the number of descriptors is very large as it 
is
> in the CoMFA grid. In other word while q2 must to go down, r2 
values
> should not mean anything of relevant for validating the QSAR 
models.
> 
> Any comment is very welcome.
> 
> Orazio
> 

Orazio Nicolotti, PhD
Research Associate
Dept. Information Studies
University of Sheffield
Western Bank
Sheffield 
S10 2TN
tel. 0114 222 2657