Practical Insights Into AI System Product Quality Evaluation
Open Access
- 18 February 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IT Professional
- Vol. 28 (1) , 81-87
Abstract
This study investigates the application of the ISO/IEC 25059 standard in the quality evaluation of artificial intelligence (AI) systems, focusing on functional suitability. Through a case study evaluating an AI system developed for analyzing images of oil platform decks, we discuss the practical nuances of evaluating functional correctness. We share information on specific properties defined to operationalize the evaluation. We emphasize the need to thoroughly discuss acceptable specification limits, especially in safety-critical contexts. Furthermore, the representativeness and sample size of the test cases should be systematically considered to reflect the profile of real-world operational conditions.Keywords
This publication has 10 references indexed in Scilit:
- Naming the Pain in machine learning-enabled systems engineeringInformation and Software Technology, 2025
- ISO/IEC quality standards for AI engineeringComputer Science Review, 2024
- An Environment for the Assessment of the Functional Suitability of AI Systems2024
- Identifying concerns when specifying machine learning-enabled systems: A perspective-based approachJournal of Systems and Software, 2024
- Connecting Research and Practice for Software Product Quality Evaluation and Certification: A Software Laboratory’s 25-Year JourneyIEEE Software, 2024
- Professional Insights into Benefits and Limitations of Implementing MLOps Principles2024
- A Meta-Summary of Challenges in Building Products with ML Components – Collecting Experiences from 4758+ Practitioners2023
- A software engineering perspective on engineering machine learning systems: State of the art and challengesJournal of Systems and Software, 2021
- What Makes Agile Software Development Agile?IEEE Transactions on Software Engineering, 2021
- Guidelines for conducting and reporting case study research in software engineeringEmpirical Software Engineering, 2008