H2O: Heterogeneity-Aware Hierarchical Orchestration for Memory-Efficient On-Device LLM Inference
- 3 November 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 25 (4) , 5393-5405
Abstract
On-device Large Language Model (LLM) inference enables private, personalized AI but faces memory constraints. Despite memory optimization efforts, scaling laws continue to increase model sizes and memory pressure. In this paper, we revisit the core memory bottlenecks in on-device LLM inference and conduct a comprehensive analysis of mainstream optimization techniques. We uncover several overlooked inefficiencies: (1) model weights, not KV caches, dominate memory usage; (2) weight sparsity remains underutilized; (3) OS-level memory behaviors cause redundancy; and (4) naive weight loading leads to excessive memory residency. To address these challenges, we propose H2O, a heterogeneity-aware hierarchical orchestration framework for memory-efficient on-device LLM inference. H2O introduces three key techniques, including hierarchical weight orchestration to reduce redundant memory retention, zero-copy I/O–compute parallelism for safe and efficient memory reuse, and heterogeneity-aware inference planning to adapt to diverse mobile hardware constraints. Extensive experimental results show that H2O reduces peak memory usage by up to 60%, eliminates out-of-memory (OOM) failures for 7 B–13 B models, and improves inference latency by 34%–94% under tight memory budgets.Keywords
Funding Information
- National Key Research and Development Program of China (2022YFF0604504)
- National Natural Science Foundation of China (62422216)
- National Natural Science Foundation of China (62320106006)
- National Natural Science Foundation of China (62502543)
- National Natural Science Foundation of China (62432004)
- National Natural Science Foundation of China (62302207)
- Key Research and Development Program of Hunan Province of China (2025JK2039)
- Central South University Innovation-Driven Research Program (2023CXQD029)
This publication has 40 references indexed in Scilit:
- Adaptive Pruning for Large Language Models With Structural Importance AwarenessIEEE Internet of Things Journal, 2026
- A Comprehensive Survey of Knowledge-Driven Deep Learning for Intelligent Wireless Network Optimization in 6GIEEE Communications Surveys & Tutorials, 2025
- NC-Load : On-Demand Program Loading and Running for Computing Sharing Among IoT DevicesIEEE Internet of Things Journal, 2024
- FedAB: Truthful Federated Learning With Auction-Based Combinatorial Multi-Armed BanditIEEE Internet of Things Journal, 2023
- Distributed Artificial Intelligence Empowered by End-Edge-Cloud Computing: A SurveyIEEE Communications Surveys & Tutorials, 2022
- Transparent Computing: Development and Current StatusChinese Journal of Electronics, 2020
- BOAT: A Block-Streaming App Execution Scheme for Lightweight IoT DevicesIEEE Internet of Things Journal, 2018
- Transparent Computing: A Promising Network Computing ParadigmComputing in Science & Engineering, 2016
- On Upper-Confidence Bound Policies for Switching Bandit Problems2011
- Multi-Armed Bandit Problems2008