Entropy estimation of very short symbolic sequences
Authors:
- 7 April 2009
- journal article
- research article
- Published by American Physical Society (APS) in Physical Review E
- Vol. 79 (4) , 046208
Abstract
While entropy per unit time is a meaningful index to quantify the dynamic features of experimental time series, its estimation is often hampered in practice by the finite length of the data. We here investigate the performance of entropy estimation procedures, relying either on block entropies or Lempel-Ziv complexity, when only very short symbolic sequences are available. Heuristic analytical arguments point at the influence of temporal correlations on the bias and statistical fluctuations, and put forward a reduced effective sequence length suitable for error estimation. Numerical studies are conducted using, as benchmarks, the wealth of different dynamic regimes generated by the family of logistic maps and stochastic evolutions generated by a Markov chain of tunable correlation time. Practical guidelines and validity criteria are proposed. For instance, block entropy leads to a dramatic overestimation for sequences of low entropy, whereas it outperforms Lempel-Ziv complexity at high entropy. As a general result, the quality of entropy estimation is sensitive to the sequence temporal correlation hence self-consistently depends on the entropy value itself, thus promoting a two-step procedure. Lempel-Ziv complexity is to be preferred in the first step and remains the best estimator for highly correlated sequences.Keywords
This publication has 37 references indexed in Scilit:
- Organization of the narrative components in autobiographical speech of anorexic adolescents: A statistical and non-linear dynamical analysisNew Ideas in Psychology, 2008
- Interpretation of the Lempel-Ziv Complexity Measure in the Context of Biomedical Signal AnalysisIEEE Transactions on Biomedical Engineering, 2006
- Estimating the Entropy Rate of Spike Trains via Lempel-Ziv ComplexityNeural Computation, 2004
- Scaling behaviour of entropy estimatesJournal of Physics A: General Physics, 2002
- Entropy and Long-Range Correlations in Literary EnglishEurophysics Letters, 1994
- Mutual information functions versus correlation functionsJournal of Statistical Physics, 1990
- Some asymptotic properties of the entropy of a stationary ergodic data source with applications to data compressionIEEE Transactions on Information Theory, 1989
- Exponential convergence to equilibrium for a class of random-walk modelsJournal of Statistical Physics, 1989
- A universal algorithm for sequential data compressionIEEE Transactions on Information Theory, 1977
- On the Complexity of Finite SequencesIEEE Transactions on Information Theory, 1976