AQUA: Attention via QUery mAgnitudes for Memory and Compute Efficient Inference in LLMs
Preprint
Abstract
The quadratic complexity of the attention mechanism remains a fundamental barrier to scaling Large Language Models (LLMs) to longer contexts, creating a critical bottleneck in both computation and memory. To address this, we introduce AQUA (Attention via QUery mAgnitudes) a novel and versatile approximation strategy that significantly reduces the cost of attention with a graceful performance trade-off. Our method operates in two phases: an efficient offline step where we compute a universal, language agnostic projection matrix via SVD on a calibration dataset, and an online inference step where we project query and key vectors and dynamically select a sparse subset of dimensions based on the query's magnitude. We provide a formal theoretical analysis of AQUA, establishing the break-even point at which it becomes more computationally efficient than standard attention. Our empirical evaluations on state-of-the-art models like Llama-3.1-8B demonstrate that a 25% reduction in the attention dot-product computation can be achieved with a statistically insignificant impact on performance across a wide range of benchmarks. We further showcase the versatility of AQUA by demonstrating its ability to synergistically accelerate existing token eviction methods like H2O and to directly reduce KV-cache memory size. By offering a controllable knob to balance efficiency and accuracy, AQUA provides a practical and powerful tool for making large-scale LLM inference more accessible and sustainable.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- A Comparative Study of Pruning Methods in Transformer-Based Time Series Forecasting2025
- BlockPruner: Fine-grained Pruning for Large Language Models2025
- Loki: Low-rank Keys for Efficient Sparse Attention2024
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark2024
- Eigen Attention: Attention in Low-Rank Space for KV Cache Compression2024
- NACL: A General and Effective KV Cache Eviction Framework for LLM at Inference Time2024
- H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models2023
- Principal component analysis: a review and recent developmentsPhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2016
- The singular value decomposition: Its computation and some applicationsIEEE Transactions on Automatic Control, 1980
- Time bounds for selectionJournal of Computer and System Sciences, 1973