From Curiosity to Competence: How World Models Interact with the Dynamics of Exploration
Preprint
Abstract
What drives an agent to explore the world while also maintaining control over the environment? From a child at play to scientists in the lab, intelligent agents must balance curiosity (the drive to seek knowledge) with competence (the drive to master and control the environment). Bridging cognitive theories of intrinsic motivation with reinforcement learning, we ask how evolving internal representations mediate the trade-off between curiosity (novelty or information gain) and competence (empowerment). We compare two model-based agents using handcrafted state abstractions (Tabular) or learning an internal world model (Dreamer). The Tabular agent shows curiosity and competence guide exploration in distinct patterns, while prioritizing both improves exploration. The Dreamer agent reveals a two-way interaction between exploration and representation learning, mirroring the developmental co-evolution of curiosity and competence. Our findings formalize adaptive exploration as a balance between pursuing the unknown and the controllable, offering insights for cognitive theories and efficient reinforcement learning.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Empowerment contributes to exploration behaviour in a creative video gameNature Human Behaviour, 2023
- A taxonomy of surprise definitionsJournal of Mathematical Psychology, 2022
- The Drive for Knowledge2022
- Challenges of real-world reinforcement learning: definitions, benchmarks and analysisMachine Learning, 2021
- Reconciling novelty and complexity through a rational analysis of curiosity.Psychological Review, 2020
- A computational analysis of the neural bases of Bayesian inferenceNeuroImage, 2015
- Information-seeking, curiosity, and attention: computational and neural mechanismsTrends in Cognitive Sciences, 2013
- An information-theoretic approach to curiosity-driven reinforcement learningTheory in Biosciences, 2012
- Learning about causes from people: Observational causal learning in 24-month-old infants.Developmental Psychology, 2012
- Finding Useful Questions: On Bayesian Diagnosticity, Probability, Impact, and Information Gain.Psychological Review, 2005