Cost-Aware Model Orchestration for LLM-based Systems
Open Access
- 27 April 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 417-425
Abstract
As modern artificial intelligence (AI) systems become more advanced and capable, they can leverage a wide range of tools and models to perform complex tasks. The task of orchestrating these models is increasingly performed by Large Language Models (LLMs) that rely on qualitative descriptions of models for decision-making. However, the descriptions provided to existing LLM-based orchestrators frequently do not reflect true model capabilities and performance characteristics, leading to suboptimal model selection, reduced task accuracy, and increased cost. In this paper, we conduct an empirical analysis of LLM-based orchestration limitations and propose a cost-aware model selection method that accounts for performance-cost trade-offs by incorporating quantitative model performance characteristics within decision-making. Initial experimental results demonstrate that our proposed method increases accuracy by 0.90%-11.92% across various evaluated tasks, achieves up to a 54% energy efficiency improvement, and reduces orchestrator model selection latency from 4.51 s to 7.2 ms.Keywords
All Related Versions
Funding Information
- Engineering and Physical Sciences Research Council (EP/V007092/1)
This publication has 5 references indexed in Scilit:
- Carbon- and Precedence-Aware Scheduling for Data Processing Clusters2025
- Energy-aware tinyML model selection on zero energy devicesInternet of Things, 2025
- The growing energy footprint of artificial intelligenceJoule, 2023
- ViperGPT: Visual Inference via Python Execution for Reasoning2023
- HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face2023