ECCOS: Efficient Capability and Cost Coordinated Scheduling for Multi-LLM Serving
Preprint
Abstract
As large language models (LLMs) are increasingly deployed as service endpoints in systems, the surge in query volume creates significant scheduling challenges. Existing scheduling frameworks mainly target at latency optimization while neglecting the capability of LLMs to serve different level of queries, which could lead to computational resource waste. This paper addresses this challenge by proposing a capability-cost coordinated scheduling framework, ECCOS, for multi-LLM serving, which explicitly constrains response quality and workload to optimize LLM inference cost. Specifically, it introduces the two-stage scheduling by designing a multi-objective predictor and a constrained optimizer. The predictor estimates both model capabilities and computational costs through training-based and retrieval-based approaches, while the optimizer determines cost-optimal assignments under quality and workload constraints. It also introduces QAServe, a dataset collected for sample-wise response quality and costs by zero-shot prompting different LLMs on knowledge QA and mathematical reasoning. Extensive experiments demonstrate that ECCOS improves success rates by 6.30% while reducing costs by 10.15% compared to existing methods, consuming less than 0.5% of LLM response time. The code is available at: https://github.com/agiresearch/ECCOS.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency2025
- Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction2024
- Efficient LLM Scheduling by Learning to Rank2024
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models2023
- Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline2023
- $S^3$: Increasing GPU Utilization during Generative Inference for Higher Throughput2023
- Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models2022
- Constrained Optimization Via Genetic AlgorithmsSIMULATION, 1994
- Constrained Optimization and Lagrange Multiplier Methods1982