Abstract
Distributed inference systems leverage hierarchical architectures spanning device, edge, and cloud to balance accuracy, latency, and resource cost. While prior work on Hierarchical Inference (HI) has primarily focused on two-tier designs, practical deployments often involve multiple models with varying capabilities. In this paper, we present a simulation-based study of a three-tier HI framework comprising two on-device models and a high-accuracy cloud model, enabling finer-grained inference decisions. We formulate the problem as an online threshold-based HI framework, where confidence scores from each model determine whether inputs are processed locally or escalated to the upper tier. An exponential weighted forecaster (EWF) strategy is used to adapt thresholds over time. We evaluate the proposed framework on CIFAR-10 and ImageNet-1K datasets under varying cost regimes. Results show that the threetier system consistently achieves near-oracle performance and outperforms all two-tier baselines by effectively handling moderate-difficulty samples at the intermediate tier. Furthermore, we observe that different feedback mechanisms yield similar average costs despite producing distinct inference acceptance patterns, indicating the presence of multiple cost-equivalent operating points. These findings demonstrate the practical advantages of three-tier HI and provide insights for designing efficient distributed ML systems.
Funding Information
  • Digital Futures and SCANIA (KTH-RPROJ-0278099)

This publication has 11 references indexed in Scilit: