Efficient Inference for Edge Large Language Models: A Survey
Authors:
Top Cited Papers
Open Access
- 1 June 2026
- journal article
- Published by Tsinghua University Press in Tsinghua Science and Technology
- Vol. 31 (3) , 1365-1380
Abstract
Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing. Their massive computational and memory requirements often necessitate cloud-based deployment, introducing challenges related to cost, latency, privacy, and network reliability. Deploying on-device LLMs alleviates these challenges, but is hindered by the severe resource constraints of edge hardware. This survey reviews efficient inference techniques for edge LLMs, with a focus on two key strategies of speculative decoding and model offloading. We categorize strategies into single-device and multi-device types, systematically analyzing the principles, recent advancements, implementations, and support within edge frameworks. Finally, we highlight the open challenges and future research directions that will advance the field of edge LLM inference.Keywords
This publication has 31 references indexed in Scilit:
- SSD Offloading for LLM Mixture-of-Experts Weights Considered Harmful in Energy EfficiencyIEEE Computer Architecture Letters, 2025
- CE-CoLLM: Efficient and Adaptive Large Language Models Through Cloud-Edge Collaboration2025
- SPEED: Speculative Pipelined Execution for Efficient Decoding2025
- Over-the-air federated learning: Status quo, open challenges, and future directionsFundamental Research, 2025
- AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration2025
- A Matching Game for LLM Layer Deployment in Heterogeneous Edge NetworksIEEE Open Journal of the Communications Society, 2025
- EdgeShard: Efficient LLM Inference via Collaborative Edge ComputingIEEE Internet of Things Journal, 2024
- Edge-LLM: A Collaborative Framework for Large Language Model Serving in Edge Computing2024
- Galaxy: A Resource-Efficient Collaborative Edge AI System for In-situ Transformer Inference2024
- PipeEdge: Pipeline Parallelism for Large-Scale Model Inference on Heterogeneous Edge Devices2022