Resource-Efficient LLM Customization on Mobile Devices Through Proxy Submodel Tuning
- 28 January 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Networking
- Vol. 34, 3687-3703
Abstract
Considering limited on-device resources, current practices are attempting to deploy a system-level mixture-of-experts (MoE)-based foundation LLM on a mobile device to serve multiple apps and support mobile intelligence. However, mobile apps are hard to customize their services that require fine-tuning adapters associated with the LLM using private in-app data. The difficulty arises due to both the limited on-device resources and the restricted control that apps have over the foundation LLM. To address this issue, in this work, we propose LiteMoE, a novel proxy submodel tuning framework that supports mobile apps to efficiently fine-tune customized adapters on devices using proxy submodels. The key technique behind LiteMoE is a post-training submodel extraction method, whereby without additional retraining, we can identify and reserve critical experts, match and merge moderate experts, to extract a lightweight and effective proxy submodel from the foundation LLM for a specific app. To further enhance scalability and adaptability, LiteMoE incorporates adapter reuse and continuous tuning mechanisms to handle multi-task requirements and evolving user preferences. We implemented a prototype of LiteMoE and evaluated it over various MoE-based LLMs and mobile computing tasks. The results show that with LiteMoE, mobile apps are able to fine-tune customized adapters on resource-limited devices, achieving 12.7% accuracy improvement and 6.6× memory reduction compared with operating the original foundation LLM.Keywords
Funding Information
- National Key Research and Development Program of China (2023YFB4502400)
- China NSF (62322206)
- China NSF (62432007)
- China NSF (62441236)
- China NSF (62132018)
- China NSF (62025204)
- China NSF (U2268204)
- China NSF (62272307)
- China NSF (62372296)
- Science and Technology Innovation 2030 –New Generation Artificial Intelligence Major (2022ZD0119100)
This publication has 47 references indexed in Scilit:
- Efficient LLM Inference on CPUs2025
- Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference2024
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- Side-Tuning: A Baseline for Network Adaptation via Additive Side Networks2020
- ZeRO: Memory optimizations Toward Training Trillion Parameter Models2020
- Searching for MobileNetV32019
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design2018
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Conflict-Aware Weighted Bipartite B-Matching and Its Application to E-CommerceIEEE Transactions on Knowledge and Data Engineering, 2016
- Fast and robust Earth Mover's Distances2009