University of Cambridge > Talks.cam > Isaac Newton Institute Seminar Series > Clustering Dynamics in Mean-Field Models of Transformers

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

Clustering Dynamics in Mean-Field Models of Transformers

Download to your calendar using vCal

If you have a question about this talk, please contact nobody.

RCL - Representing, calibrating & leveraging prediction uncertainty from statistics to machine learning

Transformers are a central architecture in modern deep learning, forming the backbone of large language models such as ChatGPT. In this talk, I will present a mathematical framework for studying how information—represented as “tokens”—evolves through the layers of such neural networks. Specifically, we consider a family of partial differential equations that describe how the distribution of tokens—modeled as particles interacting in a mean-field way—changes with depth. Numerical experiments reveal that, under certain conditions, these dynamics exhibit a metastable clustering phenomenon, where tokens group into well-separated clusters that evolve slowly over time. A rigorous analysis of this behavior uncovers a range of open questions and unexpected connections to analysis and geometry.

 

This talk is part of the Isaac Newton Institute Seminar Series series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Cambridge University Engineering Department, Rm BE4-38 (http://learning.eng.cam.ac.uk/Public/Directions) Webinar & FW11, Computer Laboratory, William Gates Building. Dept of Chemistry, Unilever Lecture Theatre

Other talks

Atmospheric Methane - rise, surge, and ongoing post-surge growth. Can we mitigate the rise, and can anything be done to reduce emissions? Title to be confirmed Machine Politics? Computers and AI in the History of Political Thought Title to be confirmed Welcome Reception and Poster Session On convective penetration of a buoyant plume into a stably stratified layer
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity