Predicting generalization of ML models.

Add to your list(s) Download to your calendar using vCal

• Vihari Piratla and Shreyas Padhy
• Wednesday 16 November 2022, 11:00-12:30
• Cambridge University Engineering Department, CBL Seminar room BE4-38..

If you have a question about this talk, please contact .

Zoom link available upon request (it is sent out on our mailing list, eng-mlg-rcc [at] lists.cam.ac.uk). Sign up to our mailing list for easier reminders.

Generalization is critical for ML applications. The standard measures of generalization using held-out splits of in-distribution (ID) data, however, tend to overestimate performance in the real-world. Moreover, it is unclear whether in-distribution performance correlates well with generalisation. Evaluating generalisation using performance on a few out-of-distribution (OOD) datasets may also fall short due to selection bias. We will begin with a discussion of large scale experiments that study the behaviour of deep learning models on OOD data. We will then discuss empirically developed generalization measures that map a trained model and training data to test error (in the real-world). These measures depend on model properties such as calibration, spectral complexity, smoothness, sensitivity to augmentations, and performance on in-domain data. We will then conclude with a discussion of theoretical developments on this thread.

Required Reading:

1: Assaying Out-Of-Distribution Generalization in Transfer Learning https://arxiv.org/pdf/2207.09239.pdf

Summary: A very large scale study of generalization. They study correlation between OOD performance and multiple model properties. They find that ID accuracy is the best predictor of OOD , but some secondary metrics can provide additional insights.

Sections to read: Abstract, Introduction

2: Methods and Analysis of The First Competition in Predicting Generalization of Deep Learning http://proceedings.mlr.press/v133/jiang21a/jiang21a.pdf

Summary: Introduces NeurIPS 2020 competition and summarizes its top-3 solutions. Contains a good motivation for the problem and big picture.

Sections to read: Abstract, Introduction, Section 3 (Solutions) up to 3.1.

This talk is part of the Machine Learning Reading Group @ CUED series.

This talk is included in these lists:

• All Talks (aka the CURE list)
• Cambridge Centre for Data-Driven Discovery (C2D3)
• Cambridge Forum of Science and Humanities
• Cambridge Language Sciences
• Cambridge University Engineering Department Talks
• Cambridge University Engineering Department, CBL Seminar room BE4-38.
• Cambridge talks
• Centre for Smart Infrastructure & Construction
• Chris Davis' list
• Computational Continuum Mechanics Group Seminars
• Featured lists
• Guy Emerson's list
• Hanchen DaDaDash
• Inference Group Journal Clubs
• Inference Group Summary
• Information Engineering Division seminar list
• Interested Talks
• ML
• Machine Learning Reading Group
• Machine Learning Reading Group @ CUED
• Machine Learning Summary
• Quantum Matter Journal Club
• Required lists for MLG
• School of Technology
• Simon Baker's List
• TQS Journal Clubs
• Trust & Technology Initiative - interesting events
• bld31
• custom
• ndk22's list
• ob366-ai4er
• rp587
• yk373's list

Note that ex-directory lists are not shown.