Deep Curriculum Reinforcement Learning for Adaptive 360° Video Streaming With Two-Stage Training
- 15 December 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Broadcasting
- Vol. 70 (2) , 441-452
Abstract
Deep reinforcement learning (DRL) has demonstrated remarkable potential within the domain of video adaptive bitrate (ABR) optimization. However, training a well-performing DRL agent in the two-tier 360 $^{\circ}$ video streaming system is non-trivial. The conventional DRL training approach fails to enable the model to start learning from simpler environments and then progressively explore more challenging ones, leading to suboptimal asymptotic performance and poor long-tail performance . In this paper, we propose a novel approach called DCRL360, which seamlessly integrates automatic curriculum learning (ACL) with DRL techniques to enable adaptive decision-making for 360 $^{\circ}$ video bitrate selection and chunk scheduling. To tackle the training issue, we introduce a structured two-stage training framework. The first stage focuses on the selection of tasks conducive to learning, guided by a newly introduced training metric called Pscore , to enhance asymptotic performance. The newly introduced metric takes into consideration multiple facets, including performance improvement potential, the risk of being forgotten, and the uncertainty of a decision, to encourage the agent to train in rewarding environments. The second stage utilizes existing rule-based techniques to identify challenging tasks for fine-tuning the model, thereby alleviating the long-tail effect. Our experimental results demonstrate that DCRL360 outperforms state-of-the-art algorithms under various network conditions -including 5G/LTE/Broadband -with a remarkable improvement of 6.51-20.86% in quality of experience (QoE), as well as a reduction in bandwidth wastage by 10.60-31.50%.Keywords
Funding Information
- National Key Research and Development Program of China (2021YFF0900503)
- National Natural Science Foundation of China (62262018, 61971382)
This publication has 37 references indexed in Scilit:
- VRCT: A Viewport Reconstruction-Based 360° Video Caching Solution for Tile-Adaptive StreamingIEEE Transactions on Broadcasting, 2023
- Improved Quality of Online Education Using Prioritized Multi-Agent Reinforcement Learning for Video Traffic SchedulingIEEE Transactions on Broadcasting, 2023
- Robust Saliency-Driven Quality Adaptation for Mobile 360-Degree Video StreamingIEEE Transactions on Mobile Computing, 2023
- QAVA: QoE-Aware Adaptive Video Bitrate Aggregation for HTTP Live Streaming Based on Smart Edge ComputingIEEE Transactions on Broadcasting, 2022
- RAM360: Robust Adaptive Multi-Layer 360$^\circ$ Video Streaming With Lyapunov OptimizationIEEE Transactions on Multimedia, 2022
- User Gaze-Driven Adaptation of Omnidirectional Video Delivery Using Spatial Tiling and Scalable Video EncodingIEEE Transactions on Broadcasting, 2022
- A Combined Field-of-View Prediction-Assisted Viewport Adaptive Delivery Scheme for 360° VideosIEEE Transactions on Broadcasting, 2021
- A Survey on Curriculum LearningIEEE Transactions on Pattern Analysis and Machine Intelligence, 2021
- DRL360: 360-degree Video Streaming with Deep Reinforcement Learning2019
- A Two-Tier System for On-Demand Streaming of 360 Degree Video Over Dynamic NetworksIEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2019