Xiangxi Mo
Research Subjects
Xiangxi Mo has a total of 6 co-authors. Here are the top co-authors by number of shared publications:
- Chen Wang
AI Cloud Platform IBM T.J. Watson Research Center Yorktown Heights NY USA
1 shared publication - Huamin Chen
IBM Research Red Hat Yorktown Heights NY Boston MA
1 shared publication - Junchen Jiang
The University of Chicago
1 shared publication Xunzhuo Liu
1 shared publication- Yue Zhu
IBM Research IBM Research Yorktown Heights NY Yorktown Heights NY
1 shared publication - Yuhan Liu
The University of Chicago
1 shared publication
Latest Publications
- Open Access
When to Reason: Semantic Router for vLLM
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems
Open AccessWhen to Reason: Semantic Router for vLLM
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also incurs significant costs in inference latency and token usage, with environmental and financial impacts, which are unnecessary for many simple prompts. We present a semantic router that classifies queries based on their reasoning requirements and selectively applies reasoning only when beneficial. Our approach achieves a 10.2 percentage point improvement in accuracy on the MMLU-Pro benchmark while reducing response latency by 47.1% and token consumption by 48.5% compared to direct inference with vLLM. These results demonstrate that semantic routing offers an effective mechanism for striking a balance between accuracy and efficiency in open-source LLM serving systems