University of Cambridge > Talks.cam > NLIP Seminar Series > Making and breaking tokenizers

  Browse A-Z

Log in

University Account External (via Google)

🔐 Log In

Information on

Subscribing to talksdetails Finding a talkdetails Adding a talkdetails Disseminating talksdetails Help and Documentationdetails

ℹ️ Information

Making and breaking tokenizers

Download to your calendar using vCal

If you have a question about this talk, please contact Suchir Salhan .

Despite massive investments in training large language models, tokenizers remain a critical but often neglected component with weaknesses that can cause wild hallucinations, bypass safety guardrails, and break downstream applications. This talk will cover:

Our recent research in automatically detecting problematic ‘glitch’ tokens in any model

Fundamental issues with pretokenizers and their design

Novel approaches to encodings and pretokenization that address some of these problems.

Speaker Bio Sander Land is a researcher at Writer, previously working at Cohere. He completed his PhD at the Department of Computer Science, University of Oxford, before undertaking a postdoc at Biomedical Engineering, King’s College London, University of London.

This talk is part of the NLIP Seminar Series series.

This talk is included in these lists:

Note that ex-directory lists are not shown.

Other lists

Room 6B, Addenbrookes Treatment Centre (Level 6) Global Food Futures The Living Technology Summit - Alibaba, China Railway, ofo, and Alipay

Other talks

Professor Philip Rosenstiel, W3 Professor of Clinical Molecular Biology, Director, Institute of Clinical Molecular Biology (IKMB), Kiel University Tissue-scale communication in cancer development Afternoon Break Formal Dinner at Selwyn College Title to be confirmed What’s my reproduction number? Life history strategies: an interdisciplinary perspective
 

© 2006-2025 Talks.cam, University of Cambridge. Contact Us | Help and Documentation | Privacy and Publicity