University of Cambridge > Talks.cam > Computational Neuroscience > The K-FAC method for neural network optimization

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹī¸ Information

The K-FAC method for neural network optimization

Download to your calendar using vCal

If you have a question about this talk, please contact Alberto Bernacchia .

Second order optimization methods have the potential to be much faster than first order methods in the deterministic case, or pre-asymptotically in the stochastic case. However traditional second order methods have proven ineffective or impractical for neural network training, due in part to the extremely high dimension of the parameter space. Kronecker-factored Approximate Curvature (K-FAC) is second-order optimization method based on a tractable approximation to the Gauss-Newton/Fisher matrix that exploits the special structure of neural network training objectives. This approximation is neither low-rank nor diagonal, but instead involves Kronecker-products, which allows for efficient estimation, storage and inversion of the curvature matrix. In this talk I will introduce the basic K-FAC method for standard MLPs and then present some more recent work in this direction, including extensions to CNNs and RNNs, both of which requires new approximations to the Fisher. For these I will provide theoretically motivated arguments, as well as empirical results which speak to their efficacy in neural network optimization.

This talk is part of the Computational Neuroscience series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Old Divinity School, St John's College, Cambridge CB2 1TP, UK https://youtu.be/keEFh3za-Eo Lecture Block, room 3

Other talks

CUSCRS and Renal Medicine Group Work The role of radiation in cancer care: a spotlight on cancers of the oesophagus, head and neck Afternoon Break Rothschild Public Lecture | Forty years of causal inference: Report of a great-grandfather Unveiling the curtain with ukubuyisa or go lata: complicated roles of museums and nation states in the cultural repatriation discourse and practice in southern Africa
 

Š 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity