Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Control
Authors:
- Haoyuan PanHPHaoyuan Pan
- Sizhao ChenSCSizhao Chen
- Zhaorui WangZWZhaorui Wang
- Tse-Tin ChanTCTse-Tin Chan
Preprint
Abstract
Ensuring timely and semantically accurate information delivery is critical in real-time wireless systems. While Age of Information (AoI) quantifies temporal freshness, Version Age of Information (VAoI) captures semantic staleness by accounting for version evolution between transmitters and receivers. Existing VAoI scheduling approaches primarily focus on minimizing average VAoI, overlooking rare but severe staleness events that can compromise reliability under stochastic packet arrivals and unreliable channels. This paper investigates both average-oriented and tail-risk-sensitive VAoI scheduling in a multi-user status update system with long-term transmission cost constraints. We first formulate the average VAoI minimization problem as a constrained Markov decision process and introduce a deep diffusion-based Soft Actor-Critic (D2SAC) algorithm. By generating actions through a diffusion-based denoising process, D2SAC enhances policy expressiveness and establishes a strong baseline for mean performance. Building on this foundation, we put forth RS-D3SAC, a risk-sensitive deep distributional diffusion-based Soft Actor-Critic algorithm. RS-D3SAC integrates a diffusion-based actor with a quantile-based distributional critic, explicitly modeling the full VAoI return distribution. This enables principled tail-risk optimization via Conditional Value-at-Risk (CVaR) while satisfying long-term transmission cost constraints. Extensive simulations show that, while D2SAC reduces average VAoI, RS-D3SAC consistently achieves substantial reductions in CVaR without sacrificing mean performance. The dominant gain in tail-risk reduction stems from the distributional critic, with the diffusion-based actor providing complementary refinement to stabilize and enrich policy decisions, highlighting their effectiveness for robust and risk-aware VAoI scheduling in multi-user wireless systems.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Optimizing Version AoI in Energy-Harvesting IoT: Model-Based and Learning-Based Approaches2025
- Offline and Distributional Reinforcement Learning for Wireless CommunicationsIEEE Communications Magazine, 2025
- Computation-Offloading Optimization for Satellite Edge Computing via Diffusion and Lyapunov-Based Deep Reinforcement LearningIEEE Internet of Things Journal, 2025
- Version Age-Optimal Cached Status Updates in a Gossiping Network With Energy Harvesting SensorIEEE Transactions on Communications, 2024
- Goal-Oriented Tensor: Beyond Age of Information Toward Semantics-Empowered Goal-Oriented CommunicationsIEEE Transactions on Communications, 2024
- Enhancing Deep Reinforcement Learning: A Tutorial on Generative Diffusion Models in Network OptimizationIEEE Communications Surveys & Tutorials, 2024
- Diffusion-Based Reinforcement Learning for Edge-Enabled AI-Generated Content ServicesIEEE Transactions on Mobile Computing, 2024
- Version Age of Information in Clustered Gossip NetworksIEEE Journal on Selected Areas in Information Theory, 2022
- Age of Information: An Introduction and SurveyIEEE Journal on Selected Areas in Communications, 2021
- Applications of Deep Reinforcement Learning in Communications and Networking: A SurveyIEEE Communications Surveys & Tutorials, 2019