Lossless Transformations and Excess Risk Bounds in Statistical Inference
Preprint
Abstract
We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss in estimating a random variable from an observed feature vector and the minimum expected loss in estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless and show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a delta-lossless transformation and give sufficient conditions for a given transformation to be universally delta-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottleneck, and deep learning, are also surveyed.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Information-Theoretic Stability and Generalization2021
- The Information Bottleneck Problem and its Applications in Machine LearningIEEE Journal on Selected Areas in Information Theory, 2020
- From the Information Bottleneck to the Privacy Funnel2014
- Strongly consistent nonparametric tests of conditional independenceStatistics & Probability Letters, 2012
- A Distribution-Free Theory of Nonparametric Regression2002
- Almost sure classification of densitiesJournal of Nonparametric Statistics, 2002
- Large deviations of divergence measures on partitionsJournal of Statistical Planning and Inference, 2001
- A Probabilistic Theory of Pattern Recognition1996
- A bound on the financial value of informationIEEE Transactions on Information Theory, 1988
- On the continuity of the error distortion function for multiple-hypothesis decisions (Corresp.)IEEE Transactions on Information Theory, 1975