Subject: Re: CCL:Re: CCL:optimization with constraints

From: <[email-masked]>
Date: Fri, 24 Mar 1995 08:19:11 EDT

Hello,

Eric Martin wrote:
>
>My understanding is that fitting a transformed function might not only
>affect the efficiency, it can also affect the answer.  The least
>squares best fit of the transformed problem will have different
>parameter values (even after the reverse transorm) than the best fit of
>the untransformed (albeit constrained) problem, since the sum of
>squared residuals of a different function is being minimized.  This can
>sometimes be rectified by applying weighted least squares.  See:
>"When, Why and How to Use Weighted Least Squares", Robert de Levie, J.
>Chem. Edu., 63, (1986) p. 10.

    This is a very good point, but it refers to a transformed _model_, 
not to constraints on the model itself (the latter being a purely 
computational issue).

A classic example is

y = A \exp (b*x)  (1)

with additive error.

In the old days (before computers), this would be changed to

\log (y) = A' + bx   (2)
which is now a linear model (in terms of the parameters A and b).

One can now use "good old linear regression" to solve this problem.
However, as Eric correctly points out, this is only (statistically) 
correct if the error is _additive_ in Eqn. (2).  Of course, if the 
error was additive in (1), this surely will not be the case in (2).  
The parameter values resulting from the 2 fits will indeed be 
different.

    One can (partially) compensate for the transformation by 
_weighting_ the residuals in fitting Eqn (2).  In general, the 
weights for a transformation f(y) are given by 

w_i = \frac{1}{f'(y_{i}}^2}

which in the case of (2) becomes 

w_i = \frac{1}{y_{i}^2}

(Of course, the above comments also assume normally distributed 
errors with equal variances, etc.)

    I never cease to be amazed by the fact that regression (and 
parameter estimation) is taught so poorly in universities - 
it's as if computers had never been invented!  There's a _huge_ 
emphasis on linear models and on normally distributed errors, and 
neglect of many important topics (errors in the independent 
variables, nonlinear models, non-normal error distributions, ....  )
Maybe it's related to the fact that beginning stats courses (where 
regression is first taught) are oriented towards the social sciences -
 where there is no such thing as a "fundamental law" and one might as 
well use a linear model.

    Some years ago, I wrote a report discussing the above and other 
problems related to nonlinear parameter estimation.  If anyone would 
like a copy, please send me an E-Mail request.

Best Regards,

W. R. Smith                        Professor
                                   Dept. of Mathematics and Statistics
                                   and School of Engineering 
                                   University of Guelph
FAX: 519-837-0221                  Guelph, Ontario
Tel: 519-824-4120, ext. 3038       CANADA N1G 2W1