Parametric Bayesian Estimation of Differential Entropy and Relative Entropy
Authors:
Open Access
Abstract
Given iid samples drawn from a distribution with known parametric form, we propose the minimization of expected Bregman divergence to form Bayesian estimates of differential entropy and relative entropy, and derive such estimators for the uniform, Gaussian, Wishart, and inverse Wishart distributions. Additionally, formulas are given for a log gamma Bregman divergence and the differential entropy and relative entropy for the Wishart and inverse Wishart. The results, as always with Bayesian estimates, depend on the accuracy of the prior parameters, but example simulations show that the performance can be substantially improved compared to maximum likelihood or state-of-the-art nonparametric estimators.Keywords
This publication has 21 references indexed in Scilit:
- Divergence Estimation for Multidimensional Densities Via $k$-Nearest-Neighbor DistancesIEEE Transactions on Information Theory, 2009
- K n -nearest neighbor estimators of entropyMathematical Methods of Statistics, 2008
- On the Estimation of Differential Entropy From Data Located on Embedded ManifoldsIEEE Transactions on Information Theory, 2007
- A Novel Biometric System for Identification and Verification of Haptic UsersIEEE Transactions on Instrumentation and Measurement, 2007
- A new class of random vector entropy estimators and its applications in testing statistical hypothesesJournal of Nonparametric Statistics, 2005
- Estimation of the entropy of a multivariate normal distributionJournal of Multivariate Analysis, 2005
- Applications of entropic spanning graphsIEEE Signal Processing Magazine, 2002
- Asymptotic theory of greedy approximations to minimal k-point random graphsIEEE Transactions on Information Theory, 1999
- Entropy expressions and their estimators for multivariate distributionsIEEE Transactions on Information Theory, 1989
- On estimation of entropy and mutual information of continuous distributionsSignal Processing, 1989