Abstract

A predictive model should ideally express its uncertainty as a probability of the output given the input. This is particularly important in high-stakes applications such as health. Predictions from black boxes come with many sources of potential uncertainty and error. Some uncertainty arises because the data does not explain perfectly the outcome. Some uncertainty comes from uncertainty on which functional form to use in the predictive model.I will discuss how analyse uncertainty from black-box classifier. The literature discusses much a quantity know as calibration. However, full uncertainty requires to go further and control the reminder, the “grouping loss”, which leads to challenging estimation problems. I’ll discuss how to estimate it, and how it connects to suboptimality gap in a decision-theory setting.Finally, I’ll do a quick tangent on table foundation models, which can be seen as a very broad way of designing complex and structured priors for predictors.