A data-augmented model routing framework for efficient LLM deployment in edge–cloud environments

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Large language model (LLM)-based program generation tasks are hindered by high computational demands. These challenges, along with high deployment costs, often pose a barrier to practical applications. To address these, we propose a novel data-augmented multi-LLM model routing approach that classifies prompts based on whether they should be processed on a weak LLM engine or a strong LLM. Experimental results show up to 16 times better efficiency compared to the existing cascaded approaches, while preserving the inference accuracy. Thus, the proposed method optimally allocates prompts across multiple LLMs, reducing computational costs while maintaining inference accuracy.
Funding Information
  • New Energy and Industrial Technology Development Organization (JPNP20017)
  • Universiti Utara Malaysia

This publication has 10 references indexed in Scilit: