How Effectively Does Metamorphic Testing Alleviate the Oracle Problem?
Top Cited Papers
- 27 September 2013
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 40 (1) , 4-22
Abstract
In software testing, something which can verify the correctness of test case execution results is called an oracle. The oracle problem occurs when either an oracle does not exist, or exists but is too expensive to be used. Metamorphic testing is a testing approach which uses metamorphic relations, properties of the software under test represented in the form of relations among inputs and outputs of multiple executions, to help verify the correctness of a program. This paper presents new empirical evidence to support this approach, which has been used to alleviate the oracle problem in various applications and to enhance several software analysis and testing techniques. It has been observed that identification of a sufficient number of appropriate metamorphic relations for testing, even by inexperienced testers, was possible with a very small amount of training. Furthermore, the cost-effectiveness of the approach could be enhanced through the use of more diverse metamorphic relations. The empirical studies presented in this paper clearly show that a small number of diverse metamorphic relations, even those identified in an ad hoc manner, had a similar fault-detection capability to a test oracle, and could thus effectively help alleviate the oracle problem.Keywords
This publication has 33 references indexed in Scilit:
- Impacts of Test Suite's Class Imbalance on Spectrum-Based Fault Localization Techniques2013
- Automated oracle creation support, or: How I learned to stop worrying about fault propagation and love mutation testing2012
- Metamorphic Testing for Web Services: Framework and a Case Study2011
- Automated metamorphic testing on the analyses of feature modelsInformation and Software Technology, 2011
- An Automatic Testing Approach for Compiler Based on Metamorphic Testing Technique2010
- Semi-Proving: An Integrated Method for Program Proving, Testing, and DebuggingIEEE Transactions on Software Engineering, 2010
- Automatic system testing of programs without test oracles2009
- An empirical study of the robustness of MacOS applications using random testingACM SIGOPS Operating Systems Review, 2007
- A practical approach to programming with assertionsIEEE Transactions on Software Engineering, 1995
- An empirical study of the reliability of UNIX utilitiesCommunications of the ACM, 1990