A Simulation Driven Evaluation of Three-tier Hierarchical Inference Learning Framework
Open Access
- 20 June 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 249-255
Abstract
Distributed inference systems leverage hierarchical architectures spanning device, edge, and cloud to balance accuracy, latency, and resource cost. While prior work on Hierarchical Inference (HI) has primarily focused on two-tier designs, practical deployments often involve multiple models with varying capabilities. In this paper, we present a simulation-based study of a three-tier HI framework comprising two on-device models and a high-accuracy cloud model, enabling finer-grained inference decisions. We formulate the problem as an online threshold-based HI framework, where confidence scores from each model determine whether inputs are processed locally or escalated to the upper tier. An exponential weighted forecaster (EWF) strategy is used to adapt thresholds over time. We evaluate the proposed framework on CIFAR-10 and ImageNet-1K datasets under varying cost regimes. Results show that the threetier system consistently achieves near-oracle performance and outperforms all two-tier baselines by effectively handling moderate-difficulty samples at the intermediate tier. Furthermore, we observe that different feedback mechanisms yield similar average costs despite producing distinct inference acceptance patterns, indicating the presence of multiple cost-equivalent operating points. These findings demonstrate the practical advantages of three-tier HI and provide insights for designing efficient distributed ML systems.Keywords
Funding Information
- Digital Futures and SCANIA (KTH-RPROJ-0278099)
This publication has 11 references indexed in Scilit:
- Hierarchical Inference at the Edge: A Batch Processing Approach2024
- Regret Bounds for Online Learning for Hierarchical Inference2024
- Getting the Best Out of Both Worlds: Algorithms for Hierarchical Inference at the EdgeIEEE Transactions on Machine Learning in Communications and Networking, 2024
- Improved Decision Module Selection for Hierarchical Inference in Resource-Constrained Edge Devices2023
- The Case for Hierarchical Deep Learning Inference at the Network Edge2023
- Data Selection Scheme for Energy Efficient Supervised Learning at IoT NodesIEEE Communications Letters, 2020
- NoScopeProceedings of the VLDB Endowment, 2017
- BranchyNet: Fast inference via early exiting from deep neural networks2016
- Prediction, Learning, and Games2006
- Rapid object detection using a boosted cascade of simple features2005