LLMSteer: Improving Long-Context LLM Inference by Steering Attention on Reused Contexts
Preprint
Abstract
As large language models (LLMs) show impressive performance on complex tasks, they still struggle with longer contextual understanding and high computational costs. To balance efficiency and quality, we introduce LLMSteer, a fine-tuning-free framework that enhances LLMs through query-independent attention steering. Tested on popular LLMs and datasets, LLMSteer narrows the performance gap with baselines by 65.9% and reduces the runtime delay by up to 4.8x compared to recent attention steering methods.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- RAGCache: Efficient Knowledge Caching for Retrieval-Augmented GenerationACM Transactions on Computer Systems, 2025
- Lost in the Middle: How Language Models Use Long ContextsTransactions of the Association for Computational Linguistics, 2024
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016