Subject: Re: CCL:neural networks problem, details

From: Adina Milac <[email-masked]>
Date: Fri, 5 Apr 2002 14:52:48 -0200 (GMT+2)

Thank you all for answering me so promptly.  Several of the messages I
have received ask me for details about the network I have used, therefore
I will give additional information about what I have done.

To be more clear, perhaps I should mention that the software I use for
neural networks is Matlab, version 4.2c.1 (if you can recommend me other
neural network software, especially running under UNIX, I would be most
grateful).

I have normed my input data in order to be able to use various transfer
functions (I have always been aware of the fact that using sigmoid 
transfer functions the network will never be able to predict an output
equal to 8). I have normed both in [-1,1] interval and in [0,1]
interval.  Results were slightly better in the latter case. I haven't
tried normalization in [-0.9;0.9], I have to try this, thanks for the
suggestion.

With the toolbox included in Matlab, I have first tried to use a simple
perceptron, but, as expected, it didn't do much (it couldn't even learn
the data properly).

After that I developed a 2-layered feed-forward network.  The  network is
able to learn but the prediction has a HUGE error if I use 7 input neurons
and still a very high error (>200% for some vectors) if I use 40 neurons
in the input layer.  In this case the problem might be with the respective
vectors because there are vectors for which the error is below 4%.

The next step was to add layers of neurons.  I have tried many
combinations of numbers of neurons in the input and hidden layer and of
transfer functions and finally I have found an "optimal" architecture of
10 (input), 15 (hidden), 1 (output).

The training algorithm I have used is an improved version of
backpropagation (Levenberg-Marquardt algorithm) because simple
backpropagation did not give satisfying results.

Practically, the code I have used to design, train and test the network is
the following (I will add some explanatory notes in case you use another
software;  if you use Matlab also and identify mistakes in the commands I
gave, please correct me):

P - matrix of 40 training vectors (each vector has 4 fields)
T - (line) matrix of target vectors

[w1,b1,w2,b2,w3,b3] = initff(P,10,'logsig',15,'logsig',T,'logsig')
        initializes a network with 10 neurons in the input layer and 15
neurons in the hidden layer; output layer has 1 neuron

tp = [50 1000 0.01]
        1000 training epochs; display error at each 50 epochs; error goal
is 0.01

[w1,b1,w2,b2,w3,b3] =
trainlm(w1,b1,'logsig',w2,b2,'logsig',w3,b3,'logsig',P,T,tp)
        the network is trained (the longest training time was about 500
epochs, it never reached 1000)

y = simuff(P,w1,b1,'logsig',w2,b2,'logsig',w3,b3,'logsig')
        simulates the system.  Instead of P I have also used the matrix of
vectors in the testing set - unseen by the network (vectors used for
prediction).
As transfer functions I have also tried tansig and purelin, but not with
signifficant differences in network performance.

************

What is even more intriguing for me is that it seems to be important the
order of the fields in the training vectors.  For example, if instead of

v1 = [f1  f2  f3  f4]

I train with

v1` = [f3  f4  f1  f2]

I get completely different results.  Should it be this way?

***********

Thank you all and good luck,

ADINA MILAC

Ph.D Student, Research Assistant
Institute of Biochemistry, Romanian Academy
Splaiul Independentei 296, 77700 Bucharest 17, Romania 
Phone: (+401).223.90.69; FAX: (+401).223.90.68
e-mail: [email-masked]