Tuning-free step-size adaptation
Authors:
- 1 March 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- No. 15206149,p. 2121-2124
Abstract
Incremental learning algorithms based on gradient descent are effective and popular in online supervised learning, reinforcement learning, signal processing, and many other application areas. An oft-noted drawback of these algorithms is that they include a step-size parameter that needs to be tuned for best performance, which may require manual intervention and significant domain knowledge or additional data. In many cases, an entire vector of step-size parameters (e.g., one for each input feature) needs to be tuned in order to attain the best performance of the algorithm. To address this, several methods have been proposed for adapting step sizes online. For example, Sutton's IDBD method can find the best vector step size for the LMS algorithm, and Schraudolph's ELK1 method, an extension of IDBD to neural networks, has proven effective on large applications, such as 3D hand tracking. However, to date all such step-size adaptation methods have included a tunable step-size parameter of their own, which we call the meta-step-size parameter. In this paper we show that the performance of existing step-size adaptation methods are strongly dependent on the choice of their meta-step-size parameter and that their meta-step-size parameter cannot be set reliably in a problem-independent way. We introduce a series of modifications and normalizations to the IDBD method that together eliminate the need to tune the meta-step-size parameter to the particular problem. We show that the resulting overall algorithm, called Autostep, performs as well or better than the existing step-size adaptation methods on a number of idealized and robot prediction problems and does not require any tuning of its meta-step-size parameter. The ideas behind Autostep are not restricted to the IDBD method and the same principles are potentially applicable to other incremental learning settings, such as reinforcement learning.Keywords
This publication has 7 references indexed in Scilit:
- Towards a robust BCI: error potentials and online learningIEEE Transactions on Neural Systems and Rehabilitation Engineering, 2006
- Scalable Techniques from Nonparametric Statistics for Real Time Robot LearningApplied Intelligence, 2002
- Parameter Adaptation in Stochastic Optimization1999
- Local gain adaptation in stochastic gradient descent1999
- Analysis of adaptive step-size SA algorithms for parameter trackingIEEE Transactions on Automatic Control, 1995
- Adaptive Algorithms and Stochastic Approximations1990
- Increased rates of convergence through learning rate adaptationNeural Networks, 1988