Improved and Generalized Upper Bounds on the Complexity of Policy Iteration
Author:
Open Access
- 1 August 2016
- journal article
- Published by Institute for Operations Research and the Management Sciences (INFORMS) in Mathematics of Operations Research
- Vol. 41 (3) , 758-774
Abstract
Given a Markov decision process (MDP) with n states and a total number m of actions, we study the number of iterations needed by policy iteration (PI) algorithms to converge to the optimal γ-discounted policy. We consider two variations of PI: Howard’s PI that changes the actions in all states with a positive advantage, and Simplex-PI that only changes the action in the state with maximal advantage. We show that Howard’s PI terminates after at most O((m/(1 − γ))log(1/(1 − γ))) iterations, improving by a factor O(log n) a result by Hansen et al. [Hansen TD, Miltersen PB, Zwick U (2013) Strategy iteration is strongly polynomial for two-player turn-based stochastic games with a constant discount factor. J. ACM 60(1):1:1–1:16.], whereas Simplex-PI terminates after at most O((n m/(1 − γ))log(1/(1 − γ))) iterations, improving by a factor O(log n) a result by Ye [Ye Y (2011) The simplex and policy-iteration methods are strongly polynomial for the Markov decision problem with a fixed discount rate. Math. Oper. Res. 36(4):593–603.]. Under some structural properties of the MDP, we then consider bounds that are independent of the discount factor γ: quantities of interest are bounds τt and τr—uniform on all states and policies—respectively, on the expected time spent in transient states and the inverse of the frequency of visits in recurrent states given that the process starts from the uniform distribution. Indeed, we show that Simplex-PI terminates after at most Õ(n3m2τtτr) iterations. This extends a recent result for deterministic MDPs by Post and Ye [Post I, Ye Y (2013) The simplex method is strongly polynomial for deterministic Markov decision processes. Khanna S, ed. Proc. 24th ACM-SIAM Sympos. Discrete Algorithms, SODA '13 (SIAM, Philadelphia), 1465–1473.] in which τt ≤ 1 and τr ≤ n; in particular it shows that Simplex-PI is strongly polynomial for a much larger class of MDPs. We explain why similar results seem hard to derive for Howard’s PI. Finally, under the additional (restrictive) assumption that the state space is partitioned in two sets, respectively, states that are transient and recurrent for all policies, we show that both Howard’s PI and Simplex-PI terminate after at most Õ(m(n2τt + nτr)) iterations.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Strategy Iteration Is Strongly Polynomial for 2-Player Turn-Based Stochastic Games with a Constant Discount FactorJournal of the ACM, 2013
- The simplex method is strongly polynomial for deterministic Markov decision processes2013
- The complexity of Policy Iteration is exponential for discounted Markov Decision Processes2012
- The Simplex and Policy-Iteration Methods Are Strongly Polynomial for the Markov Decision Problem with a Fixed Discount RateMathematics of Operations Research, 2011
- Exponential Lower Bounds for Policy Iteration2010
- Lower Bounds for Howard’s Algorithm for Finding Minimum Mean-Cost CyclesLecture Notes in Computer Science, 2010
- On the Complexity of the Policy Improvement Algorithm for Markov Decision ProcessesINFORMS Journal on Computing, 1994
- Markov Decision Processes1994
- Stochastische Matrizen1979