Subject: RE: neural networks problem

From: "Ivanciuc, Ovidiu I." <[email-masked]>
Date: Thu, 04 Apr 2002 13:30:53 -0600

Adina,
>We are trying to develop a neural network that would be able to predict
>the inhibition of some chemical compounds based on some physico-chemical
>properties of these compounds.
>The problem we face is the following:   after we have tried different
>variants of training sets, we have found that if "extreme" vectors
>(vectors in which one of the fields has either maximum or minimum
>values) are excluded from the training set, the prediction for that
>vector is extremely poor.  In other words, the network is not able to
>extrapolate the data it learns.

When using artificial neural networks (ANN) in QSAR you have to
consider that ANN are highly non-linear systems in which
extrapolation is very difficult.
Moreover, the usual transfer function of the output neuron is
the sigmoid which takes values between 0 and 1, or the hyperbolic
tangent tanh which takes values between -1 and 1.
Now suppose that in your calibration set the maximum output
(Y exp value) is 4, and in the prediction set you have a pattern
with the output 8. Now, in calibrating the ANN (optimization of the
connection weights between neurons) the Yexp=4 is mapped to +1,
which is the maximum output both for the sigmoid and tanh, and
all weights are optimized to give an output value +1 for the
input values associated with Yexp=4.
When you test the pattern with the output value Yexp=8, your output
will still be +1, since the sigmoid and tanh cannot give a larger
output, irrespective of the input values.
Therefore, you must understand that his is a mathematical limitation
of ANN with sigmoid and tanh output transfer function.

What can be done:
- scale the output values in the calibration set of molecules to a
smaller range, i.e. for tanh instead of -1 to +1 scale to -0.9
to +0.9. In this way, the ANN can offer an output value larger
than the maximum output in the calibration set.
- use a linear output transfer function, which is not bounded.
In my QSAR and QSPR experiments, I have constantly found that
a linear output function gives slightly smaller R in calibration
but better R in prediction, compared with the sigmoid and tanh.
- be aware that the ratio between the number of patterns used
in calibration (to optimize the weights) and the number of
optimizable parameters (number of connections between neurons)
must be as large as possible, but not smaller than 5 or 4.
- you have to optimize the weights with a good nonlinear
optimization algorithm; the backpropagation is bad, and usually
gives a cheap-and-dirty result. Go with a conjugate gradient
algorithm.
- always keep in mind that with QSAR models you cannot obtain good
predictions for patterns very different from those used to
calibrate the model.

Succes si o zi buna,
Ovidiu Ivanciuc
*************************

Dr. Ovidiu Ivanciuc
Sealy Center for Structural Biology,
Department of Human Biological Chemistry & Genetics,
University of Texas Medical Branch,
301 University Boulevard,
Galveston, Texas 77555-1157

---

From: "Borosy, Andras {Basi~Basel}" <[email-masked]>
Date: Thu, 04 Apr 2002 19:02:58 +0200

> 
> 2.  the network is unable to determine a function based on 
> which it could
> make the extrapolation
> 

That is the main reason. However the other regression methods suffer from that, too. If one is lucky linear relationship may valid outside the training set, if the linearity is more general.

Best wishes,

András Borosy