Finite Sample Improvement of Akaike’s Information Criterion
- 5 July 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 67 (10) , 6328-6343
Abstract
Considering the selection of frequency histograms, we propose a modification of Akaike’s Information Criterion that avoids overfitting, even when the sample size is small. We call this correction an over-penalization procedure. We emphasize that the principle of unbiased risk estimation for model selection can indeed be improved by addressing excess risk deviations in the design of the penalization procedure. On the theoretical side, we prove sharp oracle inequalities for the Kullback-Leibler divergence. These inequalities are valid with positive probability for any sample size and include the estimation of unbounded log-densities. Along the proofs, we derive several analytical lemmas related to the Kullback-Leibler divergence, as well as concentration inequalities, that are of independent interest. In a simulation study, we also demonstrate state-of-the-art performance of our over-penalization criterion for bin size selection, in particular outperforming AICc procedure.Keywords
All Related Versions
This publication has 39 references indexed in Scilit:
- Optimal upper and lower bounds for the true and empirical excess risks in heteroscedastic least-squares regressionElectronic Journal of Statistics, 2012
- A high-dimensional Wilks phenomenonProbability Theory and Related Fields, 2010
- A survey of cross-validation procedures for model selectionStatistics Surveys, 2010
- On the “degrees of freedom” of the lassoThe Annals of Statistics, 2007
- Model selection via testing: an alternative to (penalized) maximum likelihood estimatorsAnnales de l'Institut Henri Poincaré, Probabilités et Statistiques, 2006
- Optimal aggregation of classifiers in statistical learningThe Annals of Statistics, 2004
- Adaptive estimation of a quadratic functional by model selectionThe Annals of Statistics, 2000
- Mixing strategies for density estimationThe Annals of Statistics, 2000
- Approximation of Density Functions by Sequences of Exponential FamiliesThe Annals of Statistics, 1991
- Estimation of the Mean of a Multivariate Normal DistributionThe Annals of Statistics, 1981