Subject: Outliers and Neural Networks

From: "Art Owen News" <[email-masked]>
Date: Wed, 24 Nov 1993 15:03:14 -0800

Ed Jaeger <[email-masked]> wrote:
   A colleague is using neural nets to build a nonlinear model of the
   relationship between chemical structure and toxicity.  He is
   looking for diagnostic methods to seek "outliers" in his training
   set.

   Can anyone suggest such diagnostic tools or point me to where I
   might find this information?

  Here is an approach that should be useful when you are predicting 
a continuous response, using a neural network.  (It wouldn't apply
if toxicity is a 0/1 variable, but could apply if toxicity is measured
continuously by e.g. an LD50 score.)   If the net is trained by        
minimizing square error, replace the squared error criterion by
a more robust one, such as Huber's criterion.  For this you will need
a rough estimate of the range of model error that could arise in
"non-outlying" data points.  Relative to least squares Huber's methods
downweights outlying residuals.  When the weight is very small, the
point is an outlier candidate.

  Huber's criterion is of the form min( z^2, A+B|z| ) where A and
B are chosen to make the criterion continuously differentiable
in z and to make the cross over between quadratic an linear take
place at a prespecified value of |z|.  Here z is the error between
observation and network prediction.  The transition ordinarily
takes place at a value of |z| equal to some multiple of the error
standard deviation of z.  For small multiples you get behavior
like L1 methods, for large multiples it is more like L2 (least
squares).  (Details are in Huber's book "Robust Statistics").

  Minimizing the sum of Huber's criterion of the errors isn't
so hard in a neural net: the chain rule still applies, what
you pass down from the output layer changes.  Huber's criterion
is like a weighted sum of squared errors with weights equal
to  "  min'(z^2,A+B|z|)/(2z)  "  where min' just means the
derivative of the minimum.  This weight is 1 for |z| in the
quadratic range, including z=0 by continuity, and it drops
off like 1/|z| outside that range.  Points where the weight
is small can be tentatively identified as outliers.

  One can, in principle, estimate the variance of z by yet
another network, though to do it well probably takes lots
of data.

-Art Owen, Dept of Statistics, Sequoia Hall, Stanford CA 94305