Resource-rational reinforcement learning and sensorimotor causal states, and resource-rational maximiners
Author:
- 12 December 2025
- journal article
- Published by The Royal Society in Interface Focus
- Vol. 15 (5)
Abstract
We propose a new computational-level objective function for theoretical biology and theoretical neuroscience that combines: reinforcement learning, the study of learning with feedback via rewards; rate-distortion theory, a branch of information theory that deals with compressing signals to retain relevant information; and computational mechanics, the study of minimal sufficient statistics of prediction, also known as causal states. We highlight why this proposal is likely only an approximation, but is likely to be an interesting one, and propose a new algorithm for evaluating it to obtain the newly coined ‘reward-rate manifold’. The performance of real and artificial agents in partially observable environments can be newly benchmarked using these reward-rate manifolds. To that end, we calculate an example reward-rate manifold utilizing new equations reminiscent of the Blahut–Arimoto algorithm and propose a general algorithm for computing reward-rate manifolds. Finally, we describe experiments that can probe whether or not biological organisms are resource-rational reinforcement learners, using as an example maximin strategies, as bacteria have been shown to be approximate maximiners—doing their best in the worst-case environment, regardless of what is actually happening. This proposal explains why ‘good enough’ for organisms might actually be near-optimal, if viewed correctly.Keywords
All Related Versions
This publication has 49 references indexed in Scilit:
- From reinforcement learning to agency: Frameworks for understanding basal cognitionBioSystems, 2023
- Prediction and Dissipation in Nonequilibrium Molecular Sensors: Conditionally Markovian Channels Driven by Memoryful EnvironmentsBulletin of Mathematical Biology, 2020
- Predictive Rate-Distortion for Infinite-Order Markov ProcessesJournal of Statistical Physics, 2016
- Statistical signatures of structural organization: The case of long memory in renewal processesPhysics Letters A, 2016
- Exact complexity: The spectral decomposition of intrinsic computationPhysics Letters A, 2016
- Computational Mechanics of Input–Output Processes: Structured Transformations and the $\epsilon $-TransducerJournal of Statistical Physics, 2015
- Implications of rational inattentionJournal of Monetary Economics, 2003
- Computational Mechanics: Pattern and Prediction, Structure and SimplicityJournal of Statistical Physics, 2001
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998
- Optimal Adaptive Policies for Sequential Allocation ProblemsAdvances in Applied Mathematics, 1996