A2C is a special case of PPO

Authors:
  • SHShengyi Huang
    Shengyi Huang
  • AKAnssi Kanervisto
    Anssi Kanervisto
  • ARAntonin Raffin
    Antonin Raffin
  • WWWeixun Wang
    Weixun Wang
  • SOSantiago Ontañón
    Santiago Ontañón
  • RDRousslan Fernand Julien Dossa
    Rousslan Fernand Julien Dossa
Preprint
Publisher website
Google Scholar
Abstract
Advantage Actor-critic (A2C) and Proximal Policy Optimization (PPO) are popular deep reinforcement learning algorithms used for game AI in recent years. A common understanding is that A2C and PPO are separate algorithms because PPO's clipped objective appears significantly different than A2C's objective. In this paper, however, we show A2C is a special case of PPO. We present theoretical justifications and pseudocode analysis to demonstrate why. To validate our claim, we conduct an empirical experiment using \texttt{Stable-baselines3}, showing A2C and PPO produce the \textit{exact} same models when other settings are controlled.
All Related Versions