BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//Talks.cam//talks.cam.ac.uk//
X-WR-CALNAME:Talks.cam
BEGIN:VEVENT
SUMMARY:Adaptive Tokenization and Memory in Foundation Models - Edoardo Ma
 ria Ponti (University of Edinburgh)
DTSTART:20241101T120000Z
DTEND:20241101T130000Z
UID:TALK222616@talks.cam.ac.uk
CONTACT:Suchir Salhan
DESCRIPTION:\nAbstract: State-of-the-art foundation models (FMs) process i
 nformation as a sequence of internal representations\; however\, the lengt
 h of this sequence is fixed and entirely determined by tokenization. This 
 essentially decouples representation granularity from information content\
 , which exacerbates the deployment costs of FMs and narrows their “horiz
 ons” in long sequences. What if\, instead\, we could dynamically adapt t
 okenization and memory in FMs to save computation while maintaining or eve
 n enhancing performance?\n\nFirst\, I will show how we can dynamically com
 press the key-value cache of Transformers by deciding when to append or me
 rge items to memory. This offers a compromise between Transformers\, whose
  linear key-value cache growth exhausts memory space and increases latency
 \, and State Space Models\, whose finite capacity may result in forgetfuln
 ess. Secondly\, I will demonstrate how FMs can be “freed” from the tok
 enizers they are bound to by swapping them on-the-fly with arbitrary ones.
  Taking a step further\, we can even get rid of tokenizers entirely by lea
 rning end-to-end how to jointly segment and model language.\n\nCrucially\,
  this new family of FM architectures equipped with adaptive memory and tok
 enization does not require to be trained from scratch\; instead\, pre-exis
 ting open-weight FMs can be retrofitted with a negligible amount of data f
 or this purpose.\n\nBio: Edoardo M. Ponti is a Lecturer (≈ Assistant Pro
 fessor) in Natural Language Processing at the University of Edinburgh\, an
  Affiliated Lecturer at the University of Cambridge\, and a visiting profe
 ssor at NVIDIA. Previously\, he was a visiting postdoctoral scholar at Sta
 nford University and a postdoctoral fellow at Mila and McGill University i
 n Montreal. In 2021\, he obtained a PhD in computational linguistics from 
 the University of Cambridge\, St John’s College. His main research foci 
 are efficient memory and tokenization\, modular deep learning\, and comput
 ational typology. His research earned him a Google Research Faculty Award 
 and 2 Best Paper Awards at EMNLP 2021 and RepL4NLP 2019. He is a board mem
 ber and co-founder of SIGTYP\, the ACL special interest group for computat
 ional typology\, and a scholar of the European Lab for Learning and Intell
 igent Systems (ELLIS). He is a (terrible) violinist\, football player\, an
 d an aspiring practitioner of heroic viticulture.
LOCATION:Zoom link: https://cam-ac-uk.zoom.us/j/4751389294?pwd=Z2ZOSDk0eG1
 wZldVWG1GVVhrTzFIZz09
END:VEVENT
END:VCALENDAR