Abstract
On-device Large Language Model (LLM) inference enables private, personalized AI but faces memory constraints. Despite memory optimization efforts, scaling laws continue to increase model sizes and memory pressure. In this paper, we revisit the core memory bottlenecks in on-device LLM inference and conduct a comprehensive analysis of mainstream optimization techniques. We uncover several overlooked inefficiencies: (1) model weights, not KV caches, dominate memory usage; (2) weight sparsity remains underutilized; (3) OS-level memory behaviors cause redundancy; and (4) naive weight loading leads to excessive memory residency. To address these challenges, we propose H2O, a heterogeneity-aware hierarchical orchestration framework for memory-efficient on-device LLM inference. H2O introduces three key techniques, including hierarchical weight orchestration to reduce redundant memory retention, zero-copy I/O–compute parallelism for safe and efficient memory reuse, and heterogeneity-aware inference planning to adapt to diverse mobile hardware constraints. Extensive experimental results show that H2O reduces peak memory usage by up to 60%, eliminates out-of-memory (OOM) failures for 7 B–13 B models, and improves inference latency by 34%–94% under tight memory budgets.
Funding Information
  • National Key Research and Development Program of China (2022YFF0604504)
  • National Natural Science Foundation of China (62422216)
  • National Natural Science Foundation of China (62320106006)
  • National Natural Science Foundation of China (62502543)
  • National Natural Science Foundation of China (62432004)
  • National Natural Science Foundation of China (62302207)
  • Key Research and Development Program of Hunan Province of China (2025JK2039)
  • Central South University Innovation-Driven Research Program (2023CXQD029)

This publication has 40 references indexed in Scilit: