Subject: Re: CCL:Correlation Coefficient

From: Rajarshi Guha <rxg218 () psu.edu>
Date: Thu, 14 Apr 2005 17:59:24 -0400

On Thu, 2005-04-14 at 12:48 +0800, lhhu () yangtze.hku.hk wrote:
> Dear All,
> 
> I have a question about Correlation Coefficient of cross-validation. Can 
> anyone tell me how much the value of Correlation Coefficient indicate 
> the result is reliable? Certainly I know the larger the value, the more 
> reliable the result. I want to know the lowest limit of that value. I read 
> some papers that used the Correlation Coefficient value, some values are 
> lower than 0.8, is it OK for present them in journals? Or it depends on 
> different situation or topic?

There are differing views on the validity of q^2, in that high values of
q^2 do not necessarily indicate a good model in terms of predictive
ability. Some papers that discuss this topic are

@ARTICLE{q2-1,
    author = {Golbraikh, A. and Tropsha, A.},
    title = {Beware of $q^2$},
    journal = {J.~Mol.~Graph.~Model.},
    year = {2002},
    volume = {20},
    pages = {269--276},
}

@ARTICLE{q2-5,
    author = {Norinder, U.},
    title = {Single and domain made variable selection in {3D QSAR}
        applications},
    journal = {J.~Chemomet.},
    year = {1996},
    volume = {10},
    pages = {95--105},
}

@ARTICLE{q2-2,
    author = {Golbraikh, A. and Shen, M. and Xiao, Z.Y. and Xiao, Y.D.
        and Lee, K.H. and Tropsha, A.},
    title = {Rational selection of training and test sets for the
        development of validated {QSAR} models},
    journal = {J.~Comput.~Aid.~Mol.~Des.},
    year = {2003},
    volume = {17},
    pages = {241--253},
}

HTH

-------------------------------------------------------------------
Rajarshi Guha <rxg218 () psu.edu> <http://jijo.cjb.net>
GPG Fingerprint: 0CCA 8EE2 2EEB 25E2 AB04 06F7 1BB9 E634 9B87 56EE
-------------------------------------------------------------------
I'm related to people I don't relate to."
-Calvin

---

From: "Joerg K. Wegner" <wegnerj^at^informatik.uni-tuebingen.de>
Date: Thu, 14 Apr 2005 18:57:52 +0200

Hi Holly,

I would say there is no general answer and values of 0.8 or whatever 
seems suspect to me. Why? The quality depends highly on the data set 
used, even on data set with drug molecules [1].
Furthermore, if you apply an optimization algorithm to select a better 
set of features/descriptors, better induction parameters, etc. you can 
easily produce an overfitted result [2] (see references therein).
So, there is no short answer, but yes you can use the cross-validation 
result (of the outer loop) to estimate the future performance of your 
hypothesis. In other words, if you have an infinite amount of data the 
true error is the same as the experimental error. In all other cases 
with a finite number of training molecules it is not.
I would recommend to compare your result to other method based 
approaches, which is the most serious way to compare prediction rates. 
But as already said in [2] you should not use leave-one-out 
cross-validation for comparing feature selection methods, also not 
emprical selected feature sets.

[1] Wolpert, D. H. and Macready, W. G. No Free Lunch Theorems for 
Optimization IEEE Transactions on Evolutionary Computation, 1997 , 1 , 67-82
[2] Wegner, J. K. and Frvhlich, H. and Zell, A. Feature selection for 
Descriptor based Classification Models. 2. Human Intestinal Absorption 
(HIA) J. Chem. Inf. Comput. Sci., 2004 , 44 , 931-939

Kind regards, Joerg

> reliable the result. I want to know the lowest limit of that value. I 
> read some papers that used the Correlation Coefficient value, some 
> values are lower than 0.8, is it OK for present them in journals? Or it 
> depends on different situation or topic?
> 
> Thanks a lot to you all,
> 
> Holly
> 
> 
> 
> 
> 
> -= This is automatically added to each message by the mailing script =-
> To send e-mail to subscribers of CCL put the string CCL: on your 
> Subject: line
> and send your message to:  CHEMISTRY^at^ccl.net
> 
> Send your subscription/unsubscription requests to: 
> CHEMISTRY-REQUEST^at^ccl.net HOME Page: http://www.ccl.net   | Jobs Page: 
> http://www.ccl.net/jobs
> If your is mail bouncing from ccl.net domain due to spam filters, please
> use the Web based form from CCL Home Page 
> -+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> 
> 
> 
> 
> 

-- 
Dipl. Chem. Joerg K. Wegner
Center of Bioinformatics Tuebingen (ZBIT)
Department of Computer Architecture
Univ. Tuebingen, Sand 1, D-72076 Tuebingen, Germany
Phone: (+49/0) 7071 29 78970
Fax: (+49/0) 7071 29 5091
E-Mail: mailto:wegnerj^at^informatik.uni-tuebingen.de
WWW:    http://www-ra.informatik.uni-tuebingen.de
--
Never mistake motion for action.
                                     (E. Hemingway)

Never mistake action for meaningful action.
                                (Hugo Kubinyi,2004)