Efficient Reinforcement Learning from Demonstration via Bayesian Network‐Based Knowledge Extraction
Open Access
- 1 January 2021
- journal article
- research article
- Published by Wiley in Computational Intelligence and Neuroscience
- Vol. 2021 (1)
Abstract
Reinforcement learning from demonstration (RLfD) is considered to be a promising approach to improve reinforcement learning (RL) by leveraging expert demonstrations as the additional decision‐making guidance. However, most existing RLfD methods only regard demonstrations as low‐level knowledge instances under a certain task. Demonstrations are generally used to either provide additional rewards or pretrain the neural network‐based RL policy in a supervised manner, usually resulting in poor generalization capability and weak robustness performance. Considering that human knowledge is not only interpretable but also suitable for generalization, we propose to exploit the potential of demonstrations by extracting knowledge from them via Bayesian networks and develop a novel RLfD method called Reinforcement Learning from demonstration via Bayesian Network‐based Knowledge (RLBNK). The proposed RLBNK method takes advantage of node influence with the Wasserstein distance metric (NIW) algorithm to obtain abstract concepts from demonstrations and then a Bayesian network conducts knowledge learning and inference based on the abstract data set, which will yield the coarse policy with corresponding confidence. Once the coarse policy’s confidence is low, another RL‐based refine module will further optimize and fine‐tune the policy to form a (near) optimal hybrid policy. Experimental results show that the proposed RLBNK method improves the learning efficiency of corresponding baseline RL algorithms under both normal and sparse reward settings. Furthermore, we demonstrate that our RLBNK method delivers better generalization capability and robustness than baseline methods.Keywords
Funding Information
- National Natural Science Foundation of China (61703418, 61825305)
This publication has 30 references indexed in Scilit:
- Logic Tensor Networks for Semantic Image Interpretation2017
- Deep reward shaping from demonstrations2017
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- Human-level control through deep reinforcement learningNature, 2015
- Abstraction from demonstration for efficient reinforcement learning in high-dimensional domainsArtificial Intelligence, 2014
- Naive Bayes classifier for continuous variables using novel method (NBC4D) and distributions2014
- Knowledge Concepts and Categories2013
- A Survey of Discretization Techniques: Taxonomy and Empirical Analysis in Supervised LearningIEEE Transactions on Knowledge and Data Engineering, 2012
- Discretization: An Enabling TechniqueData Mining and Knowledge Discovery, 2002
- State of the Art—A Survey of Partially Observable Markov Decision Processes: Theory, Models, and AlgorithmsManagement Science, 1982