Learning From Oracle Demonstrations—A New Approach to Develop Autonomous Intersection Management Control Algorithms Based on Multiagent Deep Reinforcement Learning
Open Access
- 16 May 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 53601-53613
Abstract
Worldwide, many companies are working towards safe and innovative control systems for Autonomous Vehicles (AVs). A key component is Autonomous Intersection Management (AIM) systems, which operate at the level of traffic intersections and manage the right-of-way of AVs, improving flow and safety. AIM traditionally uses control policies based on simple rules. However, Deep Reinforcement Learning (DRL) can provide advanced control policies, with the advantage of reacting proactively and forecasting hazardous situations. The main drawback of DRL is training time, which is fast in simple tasks, but not negligible when we address real-world problems with multiple agents. Learning from Demonstrations (LfD) emerges to solve this problem, speeding up training significantly, and reducing the exploration problem. The challenge is that LfD requires an expert to extract new demonstrations. Therefore, in this paper, we propose to use an agent, previously trained by imitation learning, to act as an expert to leverage LfD. We name this new agent Oracle, and our new approach is called Learning from Oracle Demonstrations (LfOD). We have implemented this novel method over the DRL TD3 algorithm, incorporating significant changes to TD3 that allow the use of Oracle demonstrations. The complete version is called TD3fOD. The results obtained in the AIM training scenario show that TD3fOD notably improves the learning process compared with TD3, and DDPGfD, speeding up learning to 5–6 times, while the policy found offers both significantly lower variance and better learning ability. The testing scenario also shows a relevant improvement in multiple key performance metrics compared to other vehicle control techniques on AIM, such as reducing waiting time by more than 90% and significantly decreasing fuel or electricity consumption and emissions, highlighting the benefits of LfOD.Keywords
This publication has 45 references indexed in Scilit:
- Revisiting Street Intersections Using Slot-Based SystemsPLOS ONE, 2016
- Mastering the game of Go with deep neural networks and tree searchNature, 2016
- A linear programming formulation for autonomous intersection control within a dynamic traffic assignment and connected vehicle environmentTransportation Research Part C: Emerging Technologies, 2015
- Human-level control through deep reinforcement learningNature, 2015
- Batch-Light: An adaptive intelligent intersection control policy for autonomous vehicles2014
- Cooperative driving: an ant colony system for autonomous intersection managementApplied Intelligence, 2011
- Automated Intersection ControlTransportation Research Record: Journal of the Transportation Research Board, 2011
- Search-based structured predictionMachine Learning, 2009
- An Overview of Cooperative and Competitive Multiagent Learning2006
- Comparative Evaluation of Microscopic Car-Following BehaviorIEEE Transactions on Intelligent Transportation Systems, 2005