Characterizing Tests in IoT Software: Practices, Challenges and Opportunities
- 5 June 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. PP (00985589) , 1-15
Abstract
The Internet of Things (IoT) is experiencing rapid growth. Smart devices are emerging in smart homes and industrial applications, performing mission-critical tasks. Bugs in IoT software can lead to severe consequences. For example, a buggy smart lock can allow unauthorized access to a private property. Testing is a primary practice to expose software bugs and ensure software quality. However, little is known about how IoT software is tested. To bridge this gap, we conducted the first empirical study on test cases in open-source IoT software. Specifically, we evaluated the effectiveness of test cases in IoT software, explored the challenges inherent in testing IoT software, and analyzed the usage of mock objects. Our results indicate that while IoT software often contains a considerable number of tests, their effectiveness remains limited. We identified the primary challenges in testing IoT software as managing complex interactions with various external dependencies, such as other network-reliant IoT components, file systems, operating systems, and databases. We also observed that the use of mock objects in IoT software closely aligns with our identified testing challenges. This alignment demonstrates the potential of mocking as a solution to enhance test coverage and address the complexities of IoT software testing.Keywords
All Related Versions
Funding Information
- Fonds de recherche du Québec (2024-NOVA-346499 [86])
This publication has 52 references indexed in Scilit:
- An empirical study on the usage of mocking frameworks in Apache software foundationEmpirical Software Engineering, 2024
- CT-IoT: a combinatorial testing-based path selection framework for effective IoT testingEmpirical Software Engineering, 2021
- An empirical study of IoT topics in IoT developer discussions on Stack OverflowEmpirical Software Engineering, 2021
- MATTER: A tool for generating end-to-end IoT test scriptsSoftware Quality Journal, 2021
- A survey on runtime testing of dynamically adaptable and distributed systemsSoftware Quality Journal, 2021
- AUTOCON-IoT: Automated and Scalable Online Conformance Testing for IoT ApplicationsIEEE Access, 2020
- Scripted GUI testing of Android open-source apps: evolution of test code and fragility causesEmpirical Software Engineering, 2019
- IoT-TaaS: Towards a Prospective IoT Testing FrameworkIEEE Access, 2018
- Model-Based Testing as a Service for IoT Platforms2016
- An Empirical Study on the Adequacy of Testing in Open Source Projects2014