Towards Segmentation-Based Address Translation for LLM Inference
- 15 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Computer Architecture Letters
- Vol. 25 (1) , 190-193
Abstract
1As large language models (LLMs) continue to scale in both model size and context length, the efficiency of address translation has become a critical performance bottleneck during inference. Although LLM inference accesses model weights in a layer-wise pattern with substantial reuse across token generations, the expanding KV-cache footprint increasingly contends for TLB capacity, leaving fewer entries to cache weight translations. In this paper, we revisit segmentation-based address translation for model weights to bypass conventional TLB-based translation. Since model weights are read-only and not reclaimed for the lifetime of a serving instance, the segmentation-based address translation is well-suited to this access pattern. We place model weights in a virtually contiguous region backed by a physically contiguous memory, enabling segment-based translation that minimizes address translation overhead. Our evaluation shows that the proposed design improves overall IPC by 2.51× over the paging baseline by achieving near-zero translation overhead for model weights.Keywords
Funding Information
- Institute of Information & Communications Technology Planning (RS-2024-00396013)
- Institute of Information & Communications Technology Planning (RS-2024-00460762)
- Korea government
This publication has 9 references indexed in Scilit:
- Beluga: A CXL-Based Memory Architecture for Scalable and Efficient LLM KVCache ManagementProceedings of the ACM on Management of Data, 2026
- InstAttention: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference2025
- T unne L s for B ootlegging: Fully Reverse-Engineering GPU TLBs for Challenging Isolation Guarantees of NVIDIA MIG2023
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- SnakeByte: A TLB Design with Adaptive and Recursive Page Merging in GPUs2023
- Improving GPU Multi-tenancy with Page Walk Stealing2021
- Accel-Sim: An Extensible Simulation Framework for Validated GPU Modeling2020
- Efficient virtual memory for big memory servers2013
- CoLT: Coalesced Large-Reach TLBs2012