Exploration via Epistemic Value Estimation
Authors:
Open Access
- 26 June 2023
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 37 (8) , 9742-9751
Abstract
How to efficiently explore in reinforcement learning is an open problem. Many exploration algorithms employ the epistemic uncertainty of their own value predictions -- for instance to compute an exploration bonus or upper confidence bound. Unfortunately the required uncertainty is difficult to estimate in general with function approximation. We propose epistemic value estimation (EVE): a recipe that is compatible with sequential decision making and with neural network function approximators. It equips agents with a tractable posterior over all their parameters from which epistemic value uncertainty can be computed efficiently. We use the recipe to derive an epistemic Q-Learning agent and observe competitive performance on a series of benchmarks. Experiments confirm that the EVE recipe facilitates efficient exploration in hard exploration tasks.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Exploration via Epistemic Value EstimationProceedings of the AAAI Conference on Artificial Intelligence, 2023
- Mastering Atari, Go, chess and shogi by planning with a learned modelNature, 2020
- Efficient Exploration Through Bayesian Deep Q-Networks2018
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002
- A Practical Bayesian Framework for Backpropagation NetworksNeural Computation, 1992
- Asymptotic Methods in Statistical Decision Theory.Journal of the American Statistical Association, 1988
- Bayesian Decision Problems and Markov ChainsTechnometrics, 1968
- On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two SamplesBiometrika, 1933