Estimating Entropy on$m$Bins Given Fewer Than$m$Samples
Author:
- 30 August 2004
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 50 (9) , 2200-2203
Abstract
Consider a sequence p/sub N/ of discrete probability measures, supported on m/sub N/ points, and assume that we observe N independent and identically distributed (i.i.d.) samples from each p/sub N/. We demonstrate the existence of an estimator of the entropy, H(p/sub N/), which is consistent even if the ratio N/m/sub N/ is bounded (and, as a corollary, even if this ratio tends to zero, albeit at a sufficiently slow rate).Keywords
This publication has 8 references indexed in Scilit:
- Estimation of Entropy and Mutual InformationNeural Computation, 2003
- Convergence properties of functional estimates for discrete distributionsRandom Structures & Algorithms, 2001
- A Probabilistic Theory of Pattern Recognition1996
- Geometrizing Rates of Convergence, IIThe Annals of Statistics, 1991
- Geometrizing Rates of Convergence, IIIThe Annals of Statistics, 1991
- The Logarithmic Integral I1988
- Asymptotic Methods in Statistical Decision Theory1986
- A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of ObservationsThe Annals of Mathematical Statistics, 1952