NNT - Neural Network Trainer
Neural network training software for networks with arbitrarily connected neurons
You are free to download and to use the software:
(1) Download (45k) - MATLAB version, which works only in MATLAB environment
(unpack to a directory, open this directory from MATLAB, and run NNT.m)
(2) Download (177,099k) - MATLAB version, which works independently form MATLAB
(unpack to a directory and run NNT.exe)
(3) Download (535k) - NBN-2.04 written in C++, which works in Windows (unpack to a directory and run NBN 2.04.exe)
(4) Download (623k) - NBN-2.05 written in C++, which works in Windows (unpack to a directory and run NBN 2.05.exe)
(5) Download (1470k) - NBN-2.05 written in C++, which works in Windows (unpack to a directory and run NBN 2.08.exe)
The following algorithms are currently implemented in MATLAB version:
EBP - Traditional Error Back Propagation
NBN - Neuron by Neuron algorithm which is a modification of the Levenberg Marquet algorithm for arbitrarily connected neurons ACN.
SA - Self Aware algorithm which is modification of NBN. It evaluates the progression of the algorithms training and determines if the algorithm is failing to converge. If the algorithm begins to fail the weights are reset and another trial is attempted. In this situation the program displays its progress to the user as dotted red line on the display and begins again. The algorithm continues to attempt to solve the problem until either it is successful or the user cancels the process.
ESA - Enhanced Self Aware algorithm which is also modification of NBN algorithm and is used in order to increase chances for convergence. The modification was made to the Jacobian matrix in order to allow the algorithm to be much more successful in solving very difficult problems with deep local minima. It also is aware of its current solving status and will reset when necessary.
F- ESA – is another modification of NBN algorithm where an alternative method for calculating the Jacobian matrix is used. The calculation of Jacobian is unique in the sense that only feed-forward calculations are needed. This approach is then paired with ESA algorithm.
Evolutionary Gradient – newly developed algorithm, which evaluates gradients form randomly generated weight sets and use gradient information to generate new population of weights. This is a hybrid algorithm which combines the use of random populations with an approximated gradient approach. Like standard methods of evolutionary computation, the algorithm is better suited for avoiding local minima when compared to common gradient methods such as EBP. What sets the method apart is the use of an approximated gradient which is calculated with each population. By generating successive populations in the gradient direction, the algorithm is able to converge much faster than other forms of evolutionary computation. This combination of gradient and evolutionary methods essentially offers the best of both worlds.
The following algorithms are currently implemented in the C++ version:
EBP: This is EBP algorithm with traditional forward-backward computation; for EBP algorithm, it may work a little bit faster than forward-only computation. Now it is only used for standard MLP networks. EBP algorithm converges slowly, but it can be used for huge patterns training.
LM: This is LM algorithm with traditional forward-backward computation; for LM (and NBN) algorithm, the improved forward-only computation performs faster training than forward-backward computation for networks with multiple outputs. Now it is also only used for standard MLP networks. LM (and NBN) algorithm converges much faster than EBP algorithm for small and media sized patterns training.
NBN: This is NBN algorithm with forward-backward computation. NBN algorithm is developed based on LM algorithm, but it can handle arbitrarily connected neuron (ACN) networks, also, the convergence is improved [1][2].
EBP, forward-only: This is EBP algorithm with forward-only computation [3]. It can work on arbitrarily connected neuron networks.
NBN, forward-only: This is NBN algorithm with forward-only computation. It can handle arbitrarily connected neuron networks and, as mentioned above, it works faster than “NBN” algorithm, especially for networks with multiple outputs.
NBN, improved: This is a newly developed second order algorithm, implemented with forward-only computation, so it can handle arbitrarily connected neuron networks. In this algorithm, Hessian matrix is inverted only one time per iteration, so this algorithm is supposed to compute faster than LM (and NBN) algorithm which may have several times Hessian matrix inversion per iteration. The train ability (convergence) is also improved in this algorithm.
The first two algorithms only work for limited networks, while the rest of algorithms can handle arbitrarily connected neuron (ACN) networks. In spite of “NBN, improved” (need further improvement), “EBP, forward-only” and “NBN, forward-only” are recommended for training. The former one could be used for huge patterns and networks training, while the latter one is suitable for small and media sized patterns training.
Reference
[1] B. M. Wilamowski, N. Cotton, J. Hewlett, O. Kaynak, “Neural network trainer with second order learning algorithms”. Proc. International Conference on Intelligent Engineering Systems, June 29 2007-July 1 2007, pp. 127-132.
[2] Wilamowski, B.M. Cotton, N.J. Kaynak, O. Dundar, G., “Computing Gradient Vector and Jacobian Matrix in Arbitrarily Connected Neural Networks”, IEEE Trans. on Industrial Electronics, vol. 55, no. 10, pp. 3784-3790, Oct. 2008.
[3] Bogdan M. Wilamowski, Hao Yu, “Neural Network Learning without Backpropagation”. (Unpublished yet)