FaultSense: Fault Localization in Large-Scale Mixture-of-Experts Model Serving Infrastructure
Open Access
- 17 September 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 134-141
Abstract
While Mixture-of-Experts (MoE) models scale LLM serving across hundreds of GPUs, their reliance on all-to-all communication makes them susceptible to gray failures (e.g., GPU stragglers) which inflate serving latency without explicit errors, complicating fault localization. We present FaultSense, an application-layer approach that localizes faulty GPUs and communication paths in MoE model serving replicas without host instrumentation. FaultSense introduces a lightweight MoE probe model prototype alongside a two-phase hierarchical fault localization algorithm. Modeling the cluster as a GPU communication graph, it performs: (1) a global coverage that tests all GPUs and communication paths with a small set of probes to identify suspicious components, and then (2) a drill-down into suspicious components to localize faulty GPUs or communication paths. Our design reduces the number of diagnostic tests compared to individually testing all components (up to 20×) while maintaining low memory overheads.Keywords
Funding Information
- Prime Minister's Research Fellowship (PMRF)
- ZF India Technology Center
This publication has 10 references indexed in Scilit:
- FlowPulse: Catching Network Failures in ML Clusters2025
- L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis2025
- Efficient Memory Management for Large Language Model Serving with PagedAttention2023
- Adaptive Graph-Constrained Group TestingIEEE Transactions on Signal Processing, 2021
- Evolution of the Graphics Processing Unit (GPU)IEEE Micro, 2021
- Taming both predictable and unpredictable link failures for network tomography2017
- Pingmesh2015
- Identifiability of link metrics based on end-to-end path measurements2013
- Tomography-based overlay network monitoring2003
- The complexity of generalized clique coveringDiscrete Applied Mathematics, 1988