Linear combination of one-step predictive information with an external reward in an episodic policy gradient setting: a critical analysis
Authors:
Preprint
Abstract
One of the main challenges in the field of embodied artificial intelligence is the open-ended autonomous learning of complex behaviours. Our approach is to use task-independent, information-driven intrinsic motivation(s) to support task-dependent learning. The work presented here is a preliminary step in which we investigate the predictive information (the mutual information of the past and future of the sensor stream) as an intrinsic drive, ideally supporting any kind of task acquisition. Previous experiments have shown that the predictive information (PI) is a good candidate to support autonomous, open-ended learning of complex behaviours, because a maximisation of the PI corresponds to an exploration of morphology- and environment-dependent behavioural regularities. The idea is that these regularities can then be exploited in order to solve any given task. Three different experiments are presented and their results lead to the conclusion that the linear combination of the one-step PI with an external reward function is not generally recommended in an episodic policy gradient setting. Only for hard tasks a great speed-up can be achieved at the cost of an asymptotic performance lost.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- Variants of guided self-organization for robot controlTheory in Biosciences, 2011
- Deriving neural network controllers from neuro-biological data: implementation of a single-leg stick insect controllerBiological Cybernetics, 2011
- Hexapod locomotion: A nonlinear dynamical systems approach2010
- YARS: A Physical 3D Simulator for Evolving Controllers for Real RobotsLecture Notes in Computer Science, 2008
- Self-Organization, Embodiment, and Biologically Inspired RoboticsScience, 2007
- Intrinsic Motivation Systems for Autonomous Mental DevelopmentIEEE Transactions on Evolutionary Computation, 2007
- Developmental robotics, optimal artificial curiosity, creativity, music, and the fine artsConnection Science, 2006
- Maximizing Learning Progress: An Internal Reward System for Development2004
- Toward a quantitative theory of self-generated complexityInternational Journal of Theoretical Physics, 1986
- Neuronlike adaptive elements that can solve difficult learning control problemsIEEE Transactions on Systems, Man, and Cybernetics, 1983