StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
Open Access
- 24 September 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 4 (4) , 1-29
Abstract
Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregated architecture, which decouples the three RL phases (rollout, reward, and training) onto separate resources and executes them asynchronously. However, two critical data-level concerns arise: (1) asynchronous execution leads to data staleness in trajectories (the data generated by rollout) as the model parameters used in rollout may not be up to date, which impairs RL convergence; and (2) the length variation of trajectories introduces severe data skewness, leading to workload imbalance and degraded system performance. Existing systems fail to address these two concerns in a unified manner. Techniques that tightly control data staleness often constrain effective data skewness mitigation, while aggressive data skewness mitigation tends to exacerbate data staleness. As a result, systems are forced to trade off convergence for performance, or vice versa. To address this, we propose StaleFlow, an RL post-training system that jointly tackles data staleness and skewness. First, to control staleness, StaleFlow introduces a global consistency protocol that tracks the full lifecycle of each trajectory and constrains staleness. Second, to mitigate skewness, StaleFlow re-designs the RL system architecture by constructing data servers for trajectories and parameters to achieve flexible rollout coordination. Subsequently, we develop a suite of staleness-aware, throughput-oriented strategies to enhance system performance. Evaluations show that StaleFlow achieves up to 1.42-2.68× (1.18-1.91× on average) higher throughput than state-of-the-art systems, without compromising convergence. Our source code is available: https://github.com/psrl-project/psrl.Keywords
This publication has 28 references indexed in Scilit:
- Large multimodal models evaluation: a surveyScience China Information Sciences, 2025
- LLM-Based Agents for Tool Learning: A SurveyData Science and Engineering, 2025
- A Survey on Mixture of Experts in Large Language ModelsIEEE Transactions on Knowledge and Data Engineering, 2025
- The rise and potential of large language model based agents: a surveyScience China Information Sciences, 2025
- Improving Automatic Parallel Training via Balanced Memory Workload OptimizationIEEE Transactions on Knowledge and Data Engineering, 2024
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Search and Score-Based Waterfall Auction Optimization2022
- Efficient large-scale language model training on GPU clusters using megatron-LM2021
- PipeDream2019
- Multilevel Queues with Extremal PrioritiesJournal of the ACM, 1976