Shannon Entropy Rate of Hidden Markov Processes
Open Access
- 12 May 2021
- journal article
- research article
- Published by Springer Nature in Journal of Statistical Physics
- Vol. 183 (2) , 1-18
Abstract
Hidden Markov chains are widely applied statistical models of stochastic processes, from fundamental physics and chemistry to finance, health, and artificial intelligence. The hidden Markov processes they generate are notoriously complicated, however, even if the chain is finite state: no finite expression for their Shannon entropy rate exists, as the set of their predictive features is generically infinite. As such, to date one cannot make general statements about how random they are nor how structured. Here, we address the first part of this challenge by showing how to efficiently and accurately calculate their entropy rates. We also show how this method gives the minimal set of infinite predictive features. A sequel addresses the challenge’s second part on structure.Keywords
All Related Versions
Funding Information
- Army Research Laboratory (W911NF-13-1-0390, W911NF-18-1-0028)
- Foundational Questions Institute (FQXi-RFP-IPW-1902)
This publication has 39 references indexed in Scilit:
- Optimal causal inference: Estimating stored information and approximating causal architectureChaos: An Interdisciplinary Journal of Nonlinear Science, 2010
- Past-future information bottleneck in dynamical systemsPhysical Review E, 2009
- On the entropy of a hidden Markov processTheoretical Computer Science, 2008
- Stylized facts of daily return series and the hidden Markov modelJournal of Applied Econometrics, 1998
- Fractals EverywhereAmerican Journal of Physics, 1989
- An ergodic theorem for iterated mapsErgodic Theory and Dynamical Systems, 1987
- Combinatorial foundations of information theory and the calculus of probabilitiesRussian Mathematical Surveys, 1983
- Computation: Finite and Infinite Machines.The American Mathematical Monthly, 1968
- Approximations for the Entropy for Functions of Markov ChainsThe Annals of Mathematical Statistics, 1962
- Extensions of Jentzsch’s theoremTransactions of the American Mathematical Society, 1957