Abstract

Predictions for uncertain future outcomes should be calibrated in the sense that predicted probabilities for future events conform with observed event frequencies. Probabilistic predictions take the form of probability distributions over all possible values of the future outcome. If the future outcome is binary, there is a broadly agreed notion of calibration for probabilistic predictions. However, if the future outcome is more general, such as real-valued or multivariate, there are many notions of calibration that have been proposed and are considered in forecast evaluation. In this presentation, different notions of calibration will be reviewed alongside methodology to empirically assess calibration. Furthermore, the connection of calibration to proper scoring rules will be discussed.