University of Cambridge > Talks.cam > Artificial Intelligence Research Group Talks (Computer Laboratory) > Natural Experiments in NLP and Where to Find Them

Log in

University Account

External (via Google)

Information on

Subscribing to talks details

Finding a talk details

Adding a talk details

Disseminating talks details

Help and Documentation details

Natural Experiments in NLP and Where to Find Them

Download to your calendar using vCal

Pietro Lesci (University of Cambridge)
Tuesday 12 November 2024, 13:00-14:00
Lecture Theatre 2, Computer Laboratory, William Gates Building.

If you have a question about this talk, please contact Mateja Jamnik .

In training language models, training choices—such as the random seed for data ordering or the token vocabulary size—significantly influence model behaviour. Answering counterfactual questions like “How would the model perform if this instance were excluded from training?” is computationally expensive, as it requires re-training the model. Once these training configurations are set, they become fixed, creating a “natural experiment” where modifying the experimental conditions incurs high computational costs. Using econometric techniques to estimate causal effects from observational studies enables us to analyse the impact of these choices without requiring full experimental control or repeated model training. In this talk, I will present our paper, Causal Estimation of Memorisation Profiles (Best Paper Award at ACL 2024 ), which introduces a novel method based on the difference-in-differences technique from econometrics to estimate memorisation without requiring model re-training.

You can also join us on Zoom

This talk is part of the Artificial Intelligence Research Group Talks (Computer Laboratory) series.