Correctness is its own reward: bootstrapping error signals in self-guided reinforcement learning
Preprint
- 7 September 2026
- preprint
- Published by eLife Sciences Publications, Ltd in eLife preprints
- Vol. 15
Abstract
Reinforcement learning (RL) offers a compelling account of how agents learn complex behaviors by trial and error, yet RL is predicated on the existence of a reward function provided by the agent’s environment. By contrast, many skills are learned without external guidance, posing a challenge to RL’s ability to account for self-directed learning. For instance, juvenile male zebra finches first memorize and then train themselves to reproduce the song of an adult male tutor through extensive practice. This process is believed to be guided by an internally computed assessment of performance quality, though the mechanism and development of this signal remain unknown. Here, we propose that, contrary to prevailing assumptions, tutor song memorization and performance assessment are subserved by the same neural circuit, one trained to predictively cancel tutor song. To test this hypothesis, we built models of a local forebrain circuit that uses contextual premotor signals to cancel tutor song auditory input via synaptic plasticity. After learning, excitatory projection neurons signaled mismatches between the tutor song and birds’ own performance, best matching experimental data when learning involved anti-Hebbian plasticity in recurrent interneurons. We also found that learning proceeds by both sharpening error sensitivity and minimizing responses to the tutor song. Finally, the error signals produced by this model can train a simple RL agent to replicate the spectrograms of adult bird songs. These results suggest that local learning via predictive cancellation suffices for bootstrapping error signals capable of guiding self-directed learning of natural behaviors.Keywords
All Related Versions
Funding Information
- HHS | National Institutes of Health (RF1DA056376)
- HHS | National Institutes of Health (R01NS099288)
- HHS | National Institutes of Health (R01NS118424)
- Holland-Trice Foundation
This publication has 70 references indexed in Scilit:
- Vocalization modulates the mouse auditory cortex even in the absence of hearingCell Reports, 2024
- Actor-critic reinforcement learning in the songbirdCurrent Opinion in Neurobiology, 2020
- The Striatum Organizes 3D Behavior via Moment-to-Moment Action SelectionCell, 2018
- Tracking the Same Neurons across Multiple Days in Ca2+ Imaging DataCell Reports, 2017
- Motor circuits help encode auditory memories of vocal models used to guide vocal learningHearing Research, 2013
- New brain pathways found in the vocal control system of a songbirdJournal of Comparative Neurology, 2010
- Neural song preference during vocal learning in the zebra finch depends on age and stateJournal of Neurobiology, 2005
- Sparse coding of sensory inputsCurrent Opinion in Neurobiology, 2004
- A Computational Model of Birdsong Learning by Auditory Experience and Auditory Feedback1998
- Connections of vocal control nuclei in the canary telencephalonJournal of Comparative Neurology, 1982