A new algorithm for training feedforward multilayer neural networks is proposed. It is based on recursive least squares procedures and U-D factorization, which is a well-known technique in filter theory. It will be shown that due to the U-D factorization method, our algorithm requires fewer computations than the classical RLS applied to feedforward multilayer neural network training.
@article{bwmeta1.element.bwnjournal-article-amcv15i1p115bwm,
author = {Bilski, Jaros\l aw},
title = {The UD RLS algorithm for training feedforward neural networks},
journal = {International Journal of Applied Mathematics and Computer Science},
volume = {15},
year = {2005},
pages = {115-123},
zbl = {1103.68673},
language = {en},
url = {http://dml.mathdoc.fr/item/bwmeta1.element.bwnjournal-article-amcv15i1p115bwm}
}
Bilski, Jarosław. The UD RLS algorithm for training feedforward neural networks. International Journal of Applied Mathematics and Computer Science, Tome 15 (2005) pp. 115-123. http://gdmltest.u-ga.fr/item/bwmeta1.element.bwnjournal-article-amcv15i1p115bwm/
[000] Abid S., Fnaiech F. and Najim M. (2001): A fast feedforward training algorithm using a modified form of the standard backpropagation algorithm. - IEEE Trans. Neural Netw., Vol. 12, No. 2, pp. 424-434.
[001] Ampazis N. and Perantonis J. (2002): Two highly efficient second-order algorithms for training feedforward networks. - IEEE Trans. Neural Netw., Vol. 13, No. 5, pp. 1064-1074.
[002] Azimi-Sadjadi M.R. and Liou R.J. (1992): Fast learning process of multi-layer neural network using recursive least squares method. - IEEE Trans. Signal Process., Vol. 40, No. 2, pp. 443-446.
[003] Bilski J. (1995): Fast learning procedures for neural networks. - Ph.D. Thesis, AGH University of Science and Technology, (in Polish).
[004] Bilski J. and Rutkowski L. (1996): The recursive least squares method versus the backpropagation learning algorithms. - Second Conf. Neural Networks and Their Applications, Szczyrk, Poland, pp. 25-31.
[005] Bilski J. and Rutkowski L. (1998): A fast training algorithm for neural networks. - IEEE Trans. Circuits Syst. II, Vol. 45, No. 6, pp. 749-753,
[006] Bilski J. and Rutkowski L. (2003): A family of the RLS neural network learning algorithms. - Techn. Report, Dept. Comp. Eng., Technical University of Częstochowa, Poland.
[007] Bishop C.M. (1995): Neural Networks for Pattern Recognition. -Oxford: Clarendon Press. | Zbl 0868.68096
[008] Bojarczak O.S.P. and Stodolski M. (1996): Fast second-order learning algorithm for feedforward multilayer neural networks and its application. - Neural Netw. Vol. 9, No. 9, pp. 1583-1596.
[009] Chen X.- H.Y.G.- A. (1992): Efficient backpropagation learning using optimal learning rate and momentum. - Neural Netw., Vol. 10, No. 3, pp. 517-527.
[010] Joost M. and Schiffmann W. (1998): Speeding up backpropagation algorithms by using cross-entropy combined with pattern normalization. - Int. J.Uncert. Fuzz. Knowledge-Based Syst., Vol. 6, No. 2, pp. 117-126. | Zbl 1087.68617
[011] Karayiannis N.B. and Venetsanopoulos A.N. (1993): Efficient Learning Algorithms for Neural Networks (ELEANNE). - IEEE Trans. Syst. Man Cybern., Vol. 23, No. 5, pp. 1372-1383.
[012] Kitano H. (1994): Neurogenetic learning: an integrated method of designing and training neural networks using genetic algorithms. - Physica D., Vol. 75, No. 1-3, pp. 225-238. | Zbl 0860.68087
[013] Korbicz J., Obuchowicz A., Uciński D. (1994): Artificial Neural Networks. Fundamentals and Applications. - Warsaw: Akademicka Oficyna Wydawnicza PLJ, (in Polish).
[014] Lera G. and Pinzolas M. (2002): Neighbourhood based Levenberg-Marquardt algorithm for neural network training. - IEEE Trans. Neural Netw., Vol. l3, No. 5, pp. 1200-1203.
[015] Leung Ch.S., Tsoi Ah.Ch. and Chan L. W. (2001): Two regularizers for recursive least squared algorithms in feedforward multilayered neural networks. - IEEE Trans. Neural Netw., Vol. 12, No. 6, pp. 1314-1332.
[016] Moller M. (1993): A scaled conjugate gradient algorithm for fast supervised learning. - Neural Netw., Vol. 6, No. 4, pp. 525-533.
[017] Perantonis S. and Karras D. (1995): An efficient constrained learning algorithm with momentum acceleration. - Neural Netw., Vol. 8, No. 2, pp. 237-249.
[018] Rutkowski L. (1994): Adaptive Signal Processing: Theory and Applications. -Warsaw: WNT, (in Polish).
[019] Strobach P. (1990): Linear Prediction Theory - A Mathematical Basis for Adaptive Systems. - New York: Springer-Verlag. | Zbl 0727.62093
[020] Sum J., Chan L.W., Leung C.S. and Young G. (1998): Extended Kalman filter-based pruning method for recurrent neural networks. - Neural Comput., Vol. 10, No. 6, pp. 1481-1505.
[021] Sum J., Leung C., Young G.H. and Kan W. (1999): On the Kalman filtering method in neural-network training and pruning. - IEEE Trans. Neural Netw., Vol. 10, No. 1, pp. 161-166.
[022] Wellstead P.E. and Zarrop M.B. (1991): Self-Tuning Systems Control and Signal Processing. - Chichester Wiley. | Zbl 0505.93047
[023] Yao X. (1999): Evolving artificial neural networks. - Proc. IEEE, Vol. 87, No. 9, pp. 1423-1447.
[024] Zhang Y. and Li R. (1999): A fast U-D factorization-based learning algorithm with applications to nonlinear system modelling and identification. - IEEE Trans. Neural Netw., Vol. 10, No. 4, pp. 930-938.