Abstract

Conformal prediction has gained considerable attention recently since it produces forecasts that have out-of-sample calibration guarantees by construction, under the assumption of exchangeability. In this work, we study the calibration properties of conformal predictive systems, which issue sets of predictive distributions for real-valued outcomes. We demonstrate that conformal predictive systems implicitly exploit prediction methods with in-sample calibration guarantees to construct prediction sets with out-of-sample calibration guarantees. While the calibration guarantees are typically that prediction intervals derived from the predictive distributions have the correct marginal coverage, we show that this line of reasoning can be extended to stronger notions of calibration that are common in statistical forecasting theory. This allows us to take any prediction method that is calibrated in-sample, and conformalize it to obtain a conformal predictive system with out-of-sample calibration guarantees. Using this, we introduce two conformal predictive systems that exhibit stronger calibration properties than existing approaches. The first method corresponds to a binning of the data, while the second leverages isotonic distributional regression (IDR), a non-parametric distributional regression method under order constraints. We study the theoretical properties of these new conformal predictive systems, and compare their performance in a simulation experiment and two case studies. Both approaches are found to outperform existing conformal predictive systems, while conformal IDR additionally provides a natural method for quantifying epistemic uncertainty of the predictions.