Qirui Liu

In recent years, deep reinforcement learning (DRL) has made significant progress in the field of games. A prime example is AlphaZero, which, despite the formidable capabilities showcased, deters many from exploring its potential because of its demands for substantial computational resources. In this paper, we introduce BoxesZero, a computationally frugal Dots-and-Boxes agent that can achieve a high level of performance using relatively fewer computational resources. BoxesZero utilizes a novel and insightful training approach called “backward training”, which starts by training from high-reward states near the end of the game and gradually trains earlier stages of the game. It also incorporates the domain knowledge of Dots-and-Boxes, such as endgame theorems, to accelerate the Monte Carlo Tree Search (MCTS) process. Furthermore, we extend the existing endgame theorems (which only include long chains) to encompass scenarios with 1-chains and 2-chains, providing corresponding proofs, which we refer to as the extended endgame theorems. This novel agent, BoxesZero, can achieve a high level of playing strength much faster than AlphaZero, substantially improving the model’s learning efficiency. With carefully tuned parameters and limited GPU resources, BoxesZero surpasses the strongest open-source Boxes agents, PRsboxes and DabbleBoxes. Experimental results demonstrate that BoxesZero achieves an ELO rating comparable to AlphaZero in significantly less time. Furthermore, BoxesZero won the championship in the Dots-and-Boxes category of the 2024 Chinese Computer Game Competition.

Deep reinforcement learning has made remarkable strides in gaming, demonstrating prowess across perfect and imperfect information scenarios. However, its efficacy in the card game DouDiZhu has been less than optimal. DouDiZhu stands out from other games due to its multiplayer nature, blending competitive and cooperative dynamics, which translates to a complex state-action space. In 2021, DouZero emerged, leveraging Monte Carlo methods and neural networks to surpass numerous counterparts. Yet, while Monte Carlo methods boast swift training, they introduce significant variance, leading to instability during training and gameplay. To tackle this challenge, our study integrates multiple supervision heads into the neural network and enhances DouZero's architecture. Our findings reveal that our agent, DouMH, notably enhances the win rate within the same timeframe, marking agent to achieve over a 60% win rate against DouZero.

Deep reinforcement learning has achieved remarkable progress in the field of computer games, but these advancements come with significant computational resource consumption. To improve computational efficiency and reduce reliance on high computational resources, we applied two ingenious methods: reparameterization and terminal batch processing operators. By employing a specially designed replay buffer and neural networks integrated with reparameterization, and directly simulating end-game operations on the GPU, these methods not only reduce the computational burden but also ensure rapid convergence during the training process. This enhanced training framework allows for more iterations in a shorter period, thereby improving computational efficiency in self-play and training processes. We applied this technique to the game of NoGo. Experimental results indicate that, compared to the traditional AlphaZero method, the proposed approach achieves a faster increase in Elo score within the same timeframe. The agent utilizing this method secured first place among 1070 agents on the BotZone platform of Peking University.