Temporally Layered Architecture for Efficient Continuous Control
Preprint
Abstract
We present a temporally layered architecture (TLA) for temporally adaptive control with minimal energy expenditure. The TLA layers a fast and a slow policy together to achieve temporal abstraction that allows each layer to focus on a different time scale. Our design draws on the energy-saving mechanism of the human brain, which executes actions at different timescales depending on the environment's demands. We demonstrate that beyond energy saving, TLA provides many additional advantages, including persistent exploration, fewer required decisions, reduced jerk, and increased action repetition. We evaluate our method on a suite of continuous control tasks and demonstrate the significant advantages of TLA over existing methods when measured over multiple important metrics. We also introduce a multi-objective score to qualitatively assess continuous control policies and demonstrate a significantly better score for TLA. Our training algorithm uses minimal communication between the slow and fast layers to train both policies simultaneously, making it viable for future applications in distributed control.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A Survey of Reinforcement Learning Algorithms for Dynamically Varying EnvironmentsACM Computing Surveys, 2021
- Diversity-enabled sweet spots in layered architectures and speed–accuracy trade-offs in sensorimotor controlProceedings of the National Academy of Sciences, 2021
- Anticipated moments: temporal structure in attentionNature Reviews Neuroscience, 2017
- Dynamic Action Repetition for Deep Reinforcement LearningProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Temporal Prediction in lieu of Periodic StimulationThe Journal of Neuroscience, 2016
- The speed-accuracy tradeoff: history, physiology, methodology, and behaviorFrontiers in Neuroscience, 2014
- MuJoCo: A physics engine for model-based control2012
- Relationship between Jerk Cost Function and Energy Consumption during Walking2007
- Minimum jerk theory revisited2003
- Orienting attention to instants in timeNeuropsychologia, 2001