Quantitative Metrics for Performance Monitoring of Software Code Analysis Accredited Testing Laboratories
Open Access
Abstract
Modern sensors deployed in most Industry 4.0 applications are intelligent, meaning that they present sophisticated behavior, usually due to embedded software, and network connectivity capabilities. For that reason, the task of calibrating an intelligent sensor currently involves more than measuring physical quantities. As the behavior of modern sensors depends on embedded software, comprehensive assessments of such sensors necessarily demands the analysis of their embedded software. On the other hand, interlaboratory comparisons are comparative analyses of a body of labs involved in such assessments. While interlaboratory comparison is a well-established practice in fields related to physical, chemical and biological sciences, it is a recent challenge for software assessment. Establishing quantitative metrics to compare the performance of software analysis and testing accredited labs is no trivial task. Software is intangible and its requirements accommodate some ambiguity, inconsistency or information loss. Besides, software testing and analysis are highly human-dependent activities. In the present work, we investigate whether performing interlaboratory comparisons for software assessment by using quantitative performance measurement is feasible. The proposal was to evaluate the competence in software code analysis activities of each lab by using two quantitative metrics (code coverage and mutation score). Our results demonstrate the feasibility of establishing quantitative comparisons among software analysis and testing accredited laboratories. One of these rounds was registered as formal proficiency testing in the database—the first registered proficiency testing focused on code analysis.Keywords
Funding Information
- Conselho Nacional de Desenvolvimento Científico e Tecnológico (311545/2018-1, 421007/2016-8)
- Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (JCNE E-26/203.272/2017)
This publication has 49 references indexed in Scilit:
- Validating software metricsACM Transactions on Software Engineering and Methodology, 2012
- Evidence summaries: the evolution of a rapid review approachSystematic Reviews, 2012
- Proficiency Testing/External Quality Assessment: Current Challenges and Future DirectionsClinical Chemistry, 2011
- Metrology in service of society: the role of proficiency testingAccreditation and Quality Assurance, 2011
- Soft factors affecting the performance of software development teamsTeam Performance Management: An International Journal, 2011
- Measuring design complexity of semantic web ontologiesJournal of Systems and Software, 2010
- Some theoretical considerations for a suite of metrics for the integration of software componentsInformation Sciences, 2007
- A metrics suite for object oriented designIEEE Transactions on Software Engineering, 1994
- An entropy-based measure of software complexityIEEE Transactions on Software Engineering, 1992
- Evaluating software complexity measuresIEEE Transactions on Software Engineering, 1988