BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:The structure of curvature in neural networks - Alberto Bernacchia
  (MediaTek Research UK)
DTSTART:20250618T100000Z
DTEND:20250618T113000Z
UID:TALK233497@talks.cam.ac.uk
CONTACT:120952
DESCRIPTION:The curvature of the loss function plays a pivotal role in num
 erous neural network applications\, including second-order optimization\, 
 Bayesian deep learning\, iterative pruning\, and sharpness-aware minimizat
 ion. However\, the curvature matrix is typically intractable\, containing 
 O(p²) elements\, where p denotes the number of parameters. Existing tract
 able approximations—such as block-diagonal and Kronecker-factored method
 s—often suffer from inaccuracy and lack theoretical guarantees. In this 
 work\, we introduce a novel theoretical framework that precisely character
 izes the full structure of the curvature matrix by exploiting the intrinsi
 c symmetries of neural networks\, such as invariance under parameter permu
 tations. For Multi-Layer Perceptrons (MLPs)\, our approach demonstrates th
 at the global curvature can be represented using only O(d² + L²) indepen
 dent factors\, where d is the number of input/output dimensions and L is t
 he number of layers. This significantly reduces the computational complexi
 ty compared to the O(p²) elements of the full matrix. These factors can b
 e efficiently estimated\, enabling accurate curvature computations. We fur
 ther present preliminary extensions of our theory to Transformers and Recu
 rrent Neural Networks (RNNs). To assess the practical impact of our framew
 ork\, we apply second-order optimization to synthetic datasets\, achieving
  substantially faster convergence than traditional optimization methods. O
 ur findings offer new insights into the loss landscape of neural networks 
 and open avenues for the development of more efficient methodologies in de
 ep learning.
LOCATION:Cambridge University Engineering Department\, CBL Seminar room BE
 4-38.
END:VEVENT
END:VCALENDAR