Residual Context Diffusion Language Models
Authors:
- Yuezhou HuYHYuezhou Hu
- Harman SinghHSHarman Singh
- Monishwaran MaheswaranMMMonishwaran Maheswaran
- Haocheng XiHXHaocheng Xi
- Coleman HooperCHColeman Hooper
- Jintao ZhangJZJintao Zhang
- Aditya TomarATAditya Tomar
- Michael W. MahoneyMMMichael W. Mahoney
- Sewon MinSMSewon Min
- Mehrdad FarajtabarMFMehrdad Farajtabar
- Kurt KeutzerKKKurt Keutzer
- Amir GholamiAGAmir Gholami
Preprint
Abstract
Diffusion Large Language Models (dLLMs) have emerged as a promising alternative to purely autoregressive language models because they can decode multiple tokens in parallel. However, state-of-the-art block-wise dLLMs rely on a "remasking" mechanism that decodes only the most confident tokens and discards the rest, effectively wasting computation. We demonstrate that recycling computation from the discarded tokens is beneficial, as these tokens retain contextual information useful for subsequent decoding iterations. In light of this, we propose Residual Context Diffusion (RCD), a module that converts these discarded token representations into contextual residuals and injects them back for the next denoising step. RCD uses a decoupled two-stage training pipeline to bypass the memory bottlenecks associated with backpropagation. We validate our method on both long CoT reasoning (SDAR) and short CoT instruction following (LLaDA) models. We demonstrate that a standard dLLM can be efficiently converted to the RCD paradigm with merely ~1 billion tokens. RCD consistently improves frontier dLLMs by 5-10 points in accuracy with minimal extra computation overhead across a wide range of benchmarks. Notably, on the most challenging AIME tasks, RCD nearly doubles baseline accuracy and attains up to 4-5x fewer denoising steps at equivalent accuracy levels.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMsProceedings of the AAAI Conference on Artificial Intelligence, 2026
- Enhancing Auto-regressive Chain-of-Thought through Loop-Aligned Reasoning2026
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models2025
- CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillation2025
- A Careful Examination of Large Language Model Performance on Grade School Arithmetic2024
- GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers2024
- Solving Quantitative Reasoning Problems with Language Models2022
- Using the Output Embedding to Improve Language Models2017
- Deep Residual Learning for Image Recognition2016
- RooflineCommunications of the ACM, 2009