Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic Perspective
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (14) , 15625-15633
Abstract
Generative Adversarial Imitation Learning (GAIL) stands as a cornerstone approach in imitation learning. This paper investigates the gradient explosion in two types of GAIL: GAIL with deterministic policy (DE-GAIL) and GAIL with stochastic policy (ST-GAIL). We begin with the observation that the training can be highly unstable for DE-GAIL at the beginning of the training phase and end up divergence. Conversely, the ST-GAIL training trajectory remains consistent, reliably converging. To shed light on these disparities, we provide an explanation from a theoretical perspective. By establishing a probabilistic lower bound for GAIL, we demonstrate that gradient explosion is an inevitable outcome for DE-GAIL due to occasionally large expert-imitator policy disparity, whereas ST-GAIL does not have the issue with it. To substantiate our assertion, we illustrate how modifications in the reward function can mitigate the gradient explosion challenge. Finally, we propose CREDO, a simple yet effective strategy that clips the reward function during the training phase, allowing the GAIL to enjoy high data efficiency and stable trainability.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- Exploring Gradient Explosion in Generative Adversarial Imitation Learning: A Probabilistic PerspectiveProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Stochastic Policy Gradient Ascent in Reproducing Kernel Hilbert SpacesIEEE Transactions on Automatic Control, 2020
- Deterministic generative adversarial imitation learningNeurocomputing, 2020
- Efficient Training of Artificial Neural Networks for Autonomous NavigationNeural Computation, 1991