Abstract

The empirical risk minimization problem with relative entropy regularization (ERM-RER) is presented considering that the reference measure is a $\sigma$-finite measure instead of a probability measure. This generalization allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. We discuss the interplay of the regularization parameter, the reference measure, the risk measure, and the expected empirical risk induced by the solution of the ERM -RER problem, which is proved to be unique. We show that the expectation of the sensitivity is upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets. Using these tools, dataset aggregation is studied and different figures of merit to evaluate the generalization capabilities of ERM -RER are introduced. For arbitrary datasets and parameters of the ERM -RER solution, a connection between Jeffrey’s divergence, training, and test error is established. We conclude by extending the results to $f$-divergence regularization by obtaining a closed form expression for the solution under mild assumptions on the structure of the regularizer. This analytical solution is leveraged to characterize the sensitivity of the resulting supervised learning problem and we evaluate the solution for specific regularizers arising in estimation, high-dimensional statistics, and hypothesis testing.