Optimizing Attention and Cognitive Control Costs Using Temporally Layered Architectures
- 19 November 2024
- journal article
- research article
- Published by MIT Press in Neural Computation
- Vol. 36 (12) , 2734-2763
Abstract
The current reinforcement learning framework focuses exclusively on performance, often at the expense of efficiency. In contrast, biological control achieves remarkable performance while also optimizing computational energy expenditure and decision frequency. We propose a decision-bounded Markov decision process (DB-MDP) that constrains the number of decisions and computational energy available to agents in reinforcement learning environments. Our experiments demonstrate that existing reinforcement learning algorithms struggle within this framework, leading to either failure or suboptimal performance. To address this, we introduce a biologically inspired, temporally layered architecture (TLA), enabling agents to manage computational costs through two layers with distinct timescales and energy requirements. TLA achieves optimal performance in decision-bounded environments and in continuous control environments, matching state-of-the-art performance while using a fraction of the computing cost. Compared to current reinforcement learning algorithms that solely prioritize performance, our approach significantly lowers computational energy expenditure while maintaining performance. These findings establish a benchmark and pave the way for future research on energy and time-aware control.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Internal feedback in the cortical perception–action loop enables fast and accurate behaviorProceedings of the National Academy of Sciences, 2023
- Effects of Natural Scene Inversion on Visual-evoked Brain Potentials and Pupillary Responses: A Matter of Effortful Processing of Unfamiliar ConfigurationsNeuroscience, 2022
- A Survey of Reinforcement Learning Algorithms for Dynamically Varying EnvironmentsACM Computing Surveys, 2021
- Diversity-enabled sweet spots in layered architectures and speed–accuracy trade-offs in sensorimotor controlProceedings of the National Academy of Sciences, 2021
- Why Should We Add Early Exits to Neural Networks?Cognitive Computation, 2020
- Residual Reinforcement Learning for Robot Control2019
- Spinal stretch reflexes support efficient hand controlNature Neuroscience, 2019
- Scaling of sensorimotor delays in terrestrial mammalsProceedings Of The Royal Society B-Biological Sciences, 2018
- Human-level control through deep reinforcement learningNature, 2015
- Oscillatory brain responses to own names uttered by unfamiliar and familiar voicesBrain Research, 2014