The AIC Criterion and Symmetrizing the Kullback–Leibler Divergence
Authors:
- Abd-Krim SeghouaneASAbd-Krim Seghouane
- Shun-Ichi AmariSAShun-Ichi Amari
- 2 January 2007
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks
- Vol. 18 (1) , 97-106
Abstract
The Akaike information criterion (AIC) is a widely used tool for model selection. AIC is derived as an asymptotically unbiased estimator of a function used for ranking candidate models which is a variant of the Kullback-Leibler divergence between the true model and the approximating candidate model. Despite the Kullback-Leibler's computational and theoretical advantages, what can become inconvenient in model selection applications is their lack of symmetry. Simple examples can show that reversing the role of the arguments in the Kullback-Leibler divergence can yield substantially different results. In this paper, three new functions for ranking candidate models are proposed. These functions are constructed by symmetrizing the Kullback-Leibler divergence between the true model and the approximating candidate model. The operations used for symmetrizing are the average, geometric, and harmonic means. It is found that the original AIC criterion is an asymptotically unbiased estimator of these three different functions. Using one of these proposed ranking functions, an example of new bias correction to AIC is derived for univariate linear regression models. A simulation study based on polynomial regression is provided to compare the different proposed ranking functions with AIC and the new derived correction with AICcKeywords
This publication has 21 references indexed in Scilit:
- A Model Selection Algorithm for a Posteriori Probability Estimation With Neural NetworksIEEE Transactions on Neural Networks, 2005
- A Small Sample Model Selection Criterion Based on Kullback's Symmetric DivergenceIEEE Transactions on Signal Processing, 2004
- Model-order selectionIEEE Signal Processing Magazine, 2004
- A large-sample model selection criterion based on Kullback's symmetric divergenceStatistics & Probability Letters, 1999
- A CORRECTED AKAIKE INFORMATION CRITERION FOR VECTOR AUTOREGRESSIVE MODEL SELECTIONJournal of Time Series Analysis, 1993
- Regression and time series model selection in small samplesBiometrika, 1989
- The determination of optimum structures for the state space representation of multivariate stochastic processesIEEE Transactions on Automatic Control, 1982
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974
- On Information and SufficiencyThe Annals of Mathematical Statistics, 1951
- An invariant form for the prior probability in estimation problemsProceedings of the Royal Society of London. A. Mathematical and Physical Sciences, 1946