Robot Cognitive Control with a Neurophysiologically Inspired Reinforcement Learning Model
Open Access
- 1 January 2011
- journal article
- research article
- Published by Frontiers Media SA in Frontiers in Neurorobotics
- Vol. 5, 10011
Abstract
A major challenge in modern robotics is to liberate robots from controlled industrial settings, and allow them to interact with humans and changing environments in the real world. The current research attempts to determine if a neurophysiologically motivated model of cortical function in the primate can help to address this challenge. Primates are endowed with cognitive systems that allow them to maximize the feedback from their environment by learning the values of actions in diverse situations and by adjusting their behavioral parameters (i.e. cognitive control) to accommodate unexpected events. In such contexts uncertainty can arise from at least two distinct sources – expected uncertainty resulting from noise during sensory-motor interaction in a known context, and unexpected uncertainty resulting from the changing probabilistic structure of the environment. However, it is not clear how neurophysiological mechanisms of reinforcement learning and cognitive control integrate in the brain to produce efficient behavior. Based on primate neuroanatomy and neurophysiology, we propose a novel computational model for the interaction between lateral prefrontal and anterior cingulate cortex (LPFC and ACC) reconciling previous models dedicated to these two functions. We deployed the model in two robots and demonstrate that, based on adaptive regulation of a meta-parameter β that controls the exploration rate, the model can robustly deal with the two kinds of uncertainties in the real world. In addition the model could reproduce monkey behavioral performance and neurophysiological data in two problem-solving tasks. A last experiment extends this to human-robot interaction with the iCub humanoid, and novel sources of uncertainty corresponding to “cheating” by the human. The combined results provide concrete evidence for the ability of neurophysiologically inspired cognitive systems to control advanced robots in the real world.Keywords
This publication has 44 references indexed in Scilit:
- Linking language with embodied and teleological representations of action for humanoid cognitionFrontiers in Neurorobotics, 2010
- Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitationNature Neuroscience, 2009
- Where neuroscience and dynamic system theory meet autonomous robotics: A contracting basal ganglia model for action selectionNeural Networks, 2008
- Temporal Filtering of Reward Signals in the Dorsal Anterior Cingulate Cortex during a Mixed-Strategy GameJournal of Neuroscience, 2007
- Functional Specialization of the Primate Frontal Cortex during Decision Making: Figure 1.The Journal of Neuroscience, 2007
- Should I stay or should I go? How the human brain manages the trade-off between exploitation and explorationPhilosophical Transactions of the Royal Society B: Biological Sciences, 2007
- Cortical substrates for exploratory decisions in humansNature, 2006
- Anterior cingulate error‐related activity is modulated by predicted rewardEuropean Journal of Neuroscience, 2005
- Metalearning and neuromodulationNeural Networks, 2002
- A Neural Substrate of Prediction and RewardScience, 1997