Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks

Fairbank, M. & Alonso, E. (2012). Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks. Neural Computation, 24(3), pp. 607-610. doi: 10.1162/NECO_a_00248

[img]
Preview
PDF - Published Version
Download (55kB) | Preview

Abstract

The Levenberg-Marquardt (LM) learning algorithm is a popular algorithm for training neural networks; however, for large neural networks, it becomes prohibitively expensive in terms of running time and memory requirements. The most time-critical step of the algorithm is the calculation of the Gauss-Newton matrix, which is formed by multiplying two large Jacobian matrices together. We propose a method that uses back-propagation to reduce the time of this matrix-matrix multiplication. This reduces the overall asymptotic running time of the LM algorithm by a factor of the order of the number of output nodes in the neural network.

Item Type: Article
Additional Information: The article has been published in Neural Computation. © 2014 The MIT Press
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions: School of Informatics > Department of Computing
URI: http://openaccess.city.ac.uk/id/eprint/4369

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year

View more statistics