Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV Cache Compression
Preprint
Abstract
Visual Autoregressive (VAR) modeling has garnered significant attention for its innovative next-scale prediction approach, which yields substantial improvements in efficiency, scalability, and zero-shot generalization. Nevertheless, the coarse-to-fine methodology inherent in VAR results in exponential growth of the KV cache during inference, causing considerable memory consumption and computational redundancy. To address these bottlenecks, we introduce ScaleKV, a novel KV cache compression framework tailored for VAR architectures. ScaleKV leverages two critical observations: varying cache demands across transformer layers and distinct attention patterns at different scales. Based on these insights, ScaleKV categorizes transformer layers into two functional groups: drafters and refiners. Drafters exhibit dispersed attention across multiple scales, thereby requiring greater cache capacity. Conversely, refiners focus attention on the current token map to process local details, consequently necessitating substantially reduced cache capacity. ScaleKV optimizes the multi-scale inference pipeline by identifying scale-specific drafters and refiners, facilitating differentiated cache management tailored to each scale. Evaluation on the state-of-the-art text-to-image VAR model family, Infinity, demonstrates that our approach effectively reduces the required KV cache memory to 10% while preserving pixel-level fidelity.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Head-Aware KV Cache Compression for Efficient Visual Autoregressive ModelingProceedings of the AAAI Conference on Artificial Intelligence, 2026
- Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient2025
- Hash3D: Training-free Acceleration for 3D Generation2025
- Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis2025
- Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation2025
- TinyFusion: Diffusion Transformers Learned Shallow2025
- PIXART-$\Sigma $: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation2024
- MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices2024
- Cache Me if You Can: Accelerating Diffusion Models through Block Caching2024
- DeepCache: Accelerating Diffusion Models for Free2024