SHEAB: A Novel Automated Benchmarking Framework for Edge AI
Open Access
- 11 November 2025
- journal article
- review article
- Published by MDPI AG in Technologies
- Vol. 13 (11) , 515
Abstract
Edge computing is characterized by heterogeneous hardware, distributed deployment, and a need for on-site processing, which makes performance benchmarking challenging. This paper presents SHEAB (Scalable Heterogeneous Edge Automation Benchmarking), a novel framework designed to securely automate the benchmarking of Edge AI devices at scale. The proposed framework enables concurrent performance evaluation of multiple edge nodes, drastically reducing the time-to-deploy (TTD) for benchmarking tasks compared to traditional sequential methods. SHEAB’s architecture leverages containerized microservices for orchestration and result aggregation, integrated with multi-layer security (firewalls, VPN tunneling, and SSH) to ensure safe operation in untrusted network environments. We provide a detailed system design and workflow, including algorithmic pseudocode for the SHEAB process. A comprehensive comparative review of related work highlights how SHEAB advances the state-of-the-art in edge benchmarking through its combination of secure automation and scalability. We detail a real-world implementation on eleven heterogeneous edge devices, using a centralized 48-core server to coordinate benchmarks. Statistical analysis of the experimental results demonstrates a 43.74% reduction in total benchmarking time and a 1.78× speedup in benchmarking throughput using SHEAB, relative to conventional one-by-one benchmarking. We also present mathematical formulations for performance gain and discuss the implications of our results. The framework’s effectiveness is validated through the concurrent execution of standard benchmarking workloads on distributed edge nodes, with results stored in a central database for analysis. SHEAB thus represents a significant step toward efficient and reproducible Edge AI performance evaluation. Future work will extend the framework to broader workloads and further improve parallel efficiency.Keywords
Funding Information
- Széchenyi Istvan University’s Publication Support Program.
This publication has 49 references indexed in Scilit:
- EdgeMark: An automation and benchmarking system for embedded artificial intelligence toolsJournal of Systems Architecture, 2025
- Edge AI: A Taxonomy, Systematic Review and Future DirectionsCluster Computing, 2024
- Sustainable edge computing: Challenges and future directionsSoftware: Practice and Experience, 2024
- Edge AI for Internet of Medical Things: A literature reviewComputers and Electrical Engineering, 2024
- Benchmarking Container Technologies on ARM-Based Edge DevicesIEEE Access, 2023
- 5G Multi-Access Edge Computing: A Survey on Security, Dependability, and PerformanceIEEE Access, 2023
- Microservices From Cloud to Edge: An Analytical Discussion on Risks, Opportunities and EnablersIEEE Access, 2023
- A Survey on Observability of Distributed Edge & Container-Based MicroservicesIEEE Access, 2022
- Benchmarking GPU-Accelerated Edge Devices2020
- RIoTBench: An IoT benchmark for distributed stream processing systemsConcurrency and Computation: Practice and Experience, 2017