Reinforcement Learning for POMDP: Partitioned Rollout and Policy Iteration With Application to Autonomous Sequential Repair Problems
- 4 March 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Robotics and Automation Letters
- Vol. 5 (3) , 3967-3974
Abstract
In this paper we consider infinite horizon discounted partial observation dynamic programming problems with finite state and control spaces. We discuss an algorithm that uses multistep lookahead, truncated rollout with a known base policy, and a terminal cost function approximation. This algorithm is also used for policy improvement in an approximate policy iteration scheme, where successive policies are approximated by using a neural network classifier. A novel feature of our approach is that it is well suited for distributed computation through an extended belief space formulation and the use of a partitioned architecture, which is trained with multiple neural networks. We apply our methods in simulation to a class of sequential repair problems where a robot inspects and repairs a pipeline with potentially several rupture sites under partial information about the state of the pipeline.Keywords
All Related Versions
Funding Information
- National Science Foundation CAREER Award
This publication has 27 references indexed in Scilit:
- Deliberation for autonomous robots: A surveyArtificial Intelligence, 2017
- Adaptive communication in multi-robot systems using directionality of signal strengthThe International Journal of Robotics Research, 2015
- Policy Iteration Adaptive Dynamic Programming Algorithm for Discrete-Time Nonlinear SystemsIEEE Transactions on Neural Networks and Learning Systems, 2013
- A survey of point-based POMDP solversAutonomous Agents and Multi-Agent Systems, 2012
- Q-learning and policy iteration algorithms for stochastic shortest path problemsAnnals of Operations Research, 2012
- A least squares temporal difference actor–critic algorithm with applications to warehouse managementNaval Research Logistics (NRL), 2012
- Q-Learning and Enhanced Policy Iteration in Discounted Dynamic ProgrammingMathematics of Operations Research, 2012
- Experiments with Cooperative Control of Underwater RobotsThe International Journal of Robotics Research, 2009
- Rollout sampling approximate policy iterationMachine Learning, 2008
- Planning and acting in partially observable stochastic domainsArtificial Intelligence, 1998