Empirical Risk Minimization with Relative Entropy Regularization: Optimality and Sensitivity Analysis

Samir M. Perlaza, Gaetan Bisson, Iñaki Esnaola, Alain Jean-Marie, Stefano Rini

研究成果: Conference contribution同行評審

摘要

The optimality and sensitivity of the empirical risk minimization problem with relative entropy regularization (ERM-RER) are investigated for the case in which the reference is a σ-finite measure instead of a probability measure. This generalization allows for a larger degree of flexibility in the incorporation of prior knowledge over the set of models. In this setting, the interplay of the regularization parameter, the reference measure, the risk function, and the empirical risk induced by the solution of the ERM-RER problem is characterized. This characterization yields necessary and sufficient conditions for the existence of regularization parameters that achieve arbitrarily small empirical risk with arbitrarily high probability. Additionally, the sensitivity of the expected empirical risk to deviations from the solution of the ERM-RER problem is studied. Dataset-dependent and dataset-independent upper bounds on the absolute value of the sensitivity are presented. In a special case, it is shown that the expectation (with respect to the datasets) of the absolute value of the sensitivity is upper bounded, up to a constant factor, by the square root of the lautum information between the models and the datasets.

原文English
主出版物標題2022 IEEE International Symposium on Information Theory, ISIT 2022
發行者Institute of Electrical and Electronics Engineers Inc.
頁面684-689
頁數6
ISBN(電子)9781665421591
DOIs
出版狀態Published - 2022
事件2022 IEEE International Symposium on Information Theory, ISIT 2022 - Espoo, Finland
持續時間: 26 6月 20221 7月 2022

出版系列

名字IEEE International Symposium on Information Theory - Proceedings
2022-June
ISSN(列印)2157-8095

Conference

Conference2022 IEEE International Symposium on Information Theory, ISIT 2022
國家/地區Finland
城市Espoo
期間26/06/221/07/22

指紋

深入研究「Empirical Risk Minimization with Relative Entropy Regularization: Optimality and Sensitivity Analysis」主題。共同形成了獨特的指紋。

引用此