Empirical Guidelines for Deploying LLMs onto Resource-constrained Edge Devices
Open Access
- 12 August 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Design Automation of Electronic Systems
- Vol. 30 (5) , 1-58
Abstract
The scaling laws have become the de facto guidelines for designing large language models (LLMs), but they were studied under the assumption of unlimited computing resources for both training and inference. As LLMs are increasingly used as personalized intelligent assistants, their customization (i.e., learning through fine-tuning) and deployment onto resource-constrained edge devices will become more and more prevalent. An urgent but open question is how a resource-constrained computing environment would affect the design choices for a personalized LLM. We study this problem empirically in this work. In particular, we consider the tradeoffs among a number of key design factors and their intertwined impacts on learning efficiency and accuracy. The factors include the learning methods for LLM customization, the amount of personalized data used for learning customization, the types and sizes of LLMs, the compression methods of LLMs, the amount of time afforded to learn, and the difficulty levels of the target use cases. Through extensive experimentation and benchmarking, we draw a number of surprisingly insightful guidelines for deploying LLMs onto resource-constrained devices. For example, an optimal choice between parameter learning and RAG may vary depending on the difficulty of the downstream task, the longer fine-tuning time does not necessarily help the model, and a compressed LLM may be a better choice than an uncompressed LLM to learn from limited personalized data.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- A Survey on Evaluation of Large Language ModelsACM Transactions on Intelligent Systems and Technology, 2024
- Benchmarking Large Language Models in Retrieval-Augmented GenerationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the EdgeProceedings of the AAAI Conference on Artificial Intelligence, 2024
- QuadraNet: Improving High-Order Neural Interaction Efficiency with Hardware-Aware Quadratic Neural Networks2024
- Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study2023
- Parameter-efficient fine-tuning of large-scale pre-trained language modelsNature Machine Intelligence, 2023
- General Performance Score for classification problemsApplied Intelligence, 2022
- Selecting critical features for data classification based on machine learning methodsJournal of Big Data, 2020
- Evaluating human versus machine learning performance in classifying research abstractsScientometrics, 2020
- Applying Natural Language Processing and Hierarchical Machine Learning Approaches to Text Difficulty ClassificationInternational Journal of Artificial Intelligence in Education, 2020