CoDec: Prefix-Shared Decoding Kernel for LLMs
Open Access
- 18 May 2026
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Management of Data
- Vol. 4 (3) , 1-27
Abstract
Prefix-sharing among multiple prompts presents opportunities to combine the operations of the shared prefix, while attention computation in the decode stage, which becomes a critical bottleneck with increasing context lengths, is a memory-intensive process requiring heavy memory access on the key-value (KV) cache of the prefixes. Therefore, in this paper, we explore the potential of prefix-sharing in the attention computation of the decode stage. However, the tree structure of the prefix-sharing mechanism presents significant challenges for attention computation in efficiently processing shared KV cache access patterns while managing complex dependencies and balancing irregular workloads. To address the above challenges, we propose a dedicated attention kernel to co mbine the memory access of shared prefixes in the dec oding stage, namely CoDec. CoDec delivers two key innovations: a novel shared-prefix attention kernel that optimizes memory hierarchy and exploits both intra-block and inter-block parallelism, and a comprehensive workload balancing mechanism that efficiently estimates cost, divides tasks, and schedules execution. Experimental results show that CoDec achieves an average 1.9× speedup and 120.9× memory access reduction compared to the state-of-the-art FlashDecoding kernel regarding attention computation in the decode stage and 3.8× end-to-end time per output token compared to vLLM.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62325205)
- National Natural Science Foundation of China (62272215)
- National Natural Science Foundation of China (62502193)
- Natural Science Foundation of Jiangsu Province (BK20243053)
This publication has 15 references indexed in Scilit:
- Cache-Craft: Managing Chunk-Caches for Efficient Retrieval-Augmented GenerationProceedings of the ACM on Management of Data, 2025
- Squeezing Operator Performance Potential for the Ascend Architecture2025
- SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification2024
- ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference2024
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models2023
- NExT-QA: Next Phase of Question-Answering to Explaining Temporal Actions2021
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm2021
- RooflineCommunications of the ACM, 2009
- Bounds for Certain Multiprocessing AnomaliesBell System Technical Journal, 1966