City Research Online

Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks

Fairbank, M. & Alonso, E. (2012). Efficient Calculation of the Gauss-Newton Approximation of the Hessian Matrix in Neural Networks. Neural Computation, 24(3), pp. 607-610. doi: 10.1162/NECO_a_00248

Abstract

The Levenberg-Marquardt (LM) learning algorithm is a popular algorithm for training neural networks; however, for large neural networks, it becomes prohibitively expensive in terms of running time and memory requirements. The most time-critical step of the algorithm is the calculation of the Gauss-Newton matrix, which is formed by multiplying two large Jacobian matrices together. We propose a method that uses back-propagation to reduce the time of this matrix-matrix multiplication. This reduces the overall asymptotic running time of the LM algorithm by a factor of the order of the number of output nodes in the neural network.

Publication Type: Article
Additional Information: The article has been published in Neural Computation. © 2014 The MIT Press
Subjects: Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Departments: School of Science & Technology > Computer Science
SWORD Depositor:
[thumbnail of Efficient calculation.pdf]
Preview
PDF - Published Version
Download (55kB) | Preview

Export

Add to AnyAdd to TwitterAdd to FacebookAdd to LinkedinAdd to PinterestAdd to Email

Downloads

Downloads per month over past year

View more statistics

Actions (login required)

Admin Login Admin Login