A nchor : Mitigating GPU Shallow Disruptions with Decoupled Memory
Open Access
- 28 September 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 843-857
Abstract
Job failures are frequent in large-scale GPU clusters for LLM workloads, leading to significant resource wastage. The vast majority of these are shallow disruptions (e.g., software errors or updates), where only the worker process crashes while the underlying GPU and OS kernel remain intact. Existing recovery systems, however, are failure-agnostic, relying on heavyweight checkpointing that forces a full, slow state reload even when the data could have survived on the GPU. In this paper, we propose Anchor, a system that avoids this overhead by decoupling memory ownership from the failure-prone worker process. Anchor uses a daemon process to manage GPU memory. Leveraging the GPU's native IPC memory mechanism, Anchor enables a restarted worker to remap and reuse the daemon-managed memory with no overhead on the normal execution path. To make this idea practical for real-world ML applications, Anchor introduces an access-path-based identity resolution mechanism to remap logical tensors to their physical memory objects after restart. Anchor also provides lightweight, application-aware consistency protocols to ensure data integrity with little overhead. We integrate Anchor into DeepSpeed and vLLM. Our evaluation demonstrates recovery acceleration, with recovery time reduced by 60.7% for inference and 26.6% for training on average. For Qwen3-235B inference, Anchor reduces the recovery time from 143.78 s to 27.85 s.Funding Information
- National Key R&D Program of China (2024YFB4505201)
- National Natural Science Foundation of China (U22B2023)
- National Natural Science Foundation of China (62572296)
- National Natural Science Foundation of China (62332011)
- Beijing Natural Science Foundation (L242016)
This publication has 20 references indexed in Scilit:
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature, 2025
- Unified Collective Communication: A Unified Library for CPU, GPU, and DPU CollectivesIEEE Micro, 2025
- GMLake: Efficient and Transparent GPU Memory Defragmentation for Large-scale DNN Training with Virtual Memory Stitching2024
- GEMINI: Fast Failure Recovery in Distributed Training with In-Memory Checkpoints2023
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Efficient checkpoint/Restart of CUDA applicationsParallel Computing, 2023
- Accelerating Deep Learning Using Interconnect-Aware UCX Communication for MPI CollectivesIEEE Micro, 2022
- DeepSpeed2020
- DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models2020
- Diskless checkpointingIEEE Transactions on Parallel and Distributed Systems, 1998