Large-Scale Traffic Signal Control Based on Multi-Agent Q-Learning and Pressure
Open Access
- 21 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 12, 1092-1101
Abstract
Traffic congestion has become an increasingly severe problem and needs to be solved urgently. Reinforcement learning (RL) is an advanced data-driven approach for large-scale adaptive traffic signal control (ATSC) in the complex urban traffic network. The decentralized multi-agent RL (MARL) framework is feasible for multi-intersection scenarios. However, the environment becomes partially observable and nonstationary without effective communication among agents from the perspective of each local agent. Previous studies solve this problem by improving the observation of each local agent and using a spatial discount factor. In this paper, we are motivated by achieving coordination among intersections in the RL settings and better performance in ATSC and solve the problem of traffic signal control at multiple intersections in large-scale networks based on RL techniques. Pressure is used as an indicator to represent the traffic conditions of an intersection considering its neighbor intersections. A new reward function with pressure and vehicle waiting time is designed to achieve coordination among intersections. The above three control policies are innovatively applied to a multi-agent Q-learning method for ATSC at multiple intersections. Experiments are carried out on a large synthetic traffic grid under simulated peak-hour traffic dynamics. Results demonstrate that our approach outperforms the state-of-the-art decentralized MARL algorithms in average queue length and intersection delay.Keywords
Funding Information
- National Natural Science Foundation of China (61903229)
- Natural Science Foundation of Shandong Province (ZR2019BF004)
This publication has 21 references indexed in Scilit:
- IntelliLight2018
- Adaptive traffic signal control with actor-critic methods in a real-world traffic network with different traffic disruption eventsTransportation Research Part C: Emerging Technologies, 2017
- Human-level control through deep reinforcement learningNature, 2015
- Max pressure control of a network of signalized intersectionsTransportation Research Part C: Emerging Technologies, 2013
- Self-Organizing Traffic Lights: A Realistic Simulation2013
- Reinforcement Learning With Function Approximation for Traffic Signal ControlIEEE Transactions on Intelligent Transportation Systems, 2010
- Algorithms for Reinforcement LearningSynthesis Lectures on Artificial Intelligence and Machine Learning, 2010
- Reinforcement Learning: An IntroductionIEEE Transactions on Neural Networks, 1998
- Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents1993
- Q-learningMachine Learning, 1992