Bruno E. R. Garcia

Software testing through fuzzing has gained widespread adoption for discovering security vulnerabilities, yet questions remain about its effectiveness in detecting subtle behavioral faults. This paper presents an empirical evaluation investigating the intersection of fuzzing and mutation testing, specifically examining how well fuzz targets perform when evaluated through mutation analysis. We conducted a systematic study using Bitcoin Core as our subject system, analyzing 10 different fuzz targets across various modules and evaluating their ability to detect 726 generated mutants. Our methodology involved executing fuzz targets with existing seed corpora and measuring mutation scores both with and without assertion statements to understand the role of explicit oracles in fault detection. Our findings reveal that contrary to previous studies suggesting fuzzing’s limited effectiveness in mutation testing, several fuzz targets achieved high mutation scores, with two targets reaching 100% mutant detection rates. We identified three key design patterns that significantly enhance mutant detection capabilities: (1) round-trip testing approaches that verify data integrity through serialization-deserialization cycles, (2) mathematical oracles that implement exact behavioral verification through redundant calculations, and (3) metamorphic relations that validate expected relationships between inputs and outputs. Our analysis demonstrates a positive correlation between assertion density in fuzz targets and mutation scores, with assertion removal causing substantial drops in detection rates across all targets. The study contributes empirical evidence that well-designed fuzz targets can effectively detect subtle behavioral faults beyond traditional crash-based vulnerabilities. Our results suggest that incorporating explicit oracles, metamorphic properties, and roundtrip verification mechanisms into fuzz target design significantly improves their mutation testing performance. These findings have practical implications for improving fuzzing methodologies and developing more comprehensive automated testing strategies for safety-critical software systems.

Peer-to-peer (P2P) systems, characterized by their decentralized architecture, have become crucial to various applications such as file sharing, content distribution, and blockchain technologies. Due to their dynamic and heterogeneous nature, ensuring their reliability and robustness presents challenges. Such an issue has motivated researchers to develop testing techniques for these systems. Despite advances in research about software testing for distributed systems, there is a lack of data about their applicability and which features from P2P systems are highlighted by the proposed testing approaches. This paper provides a comprehensive overview of the state-of-the-art software testing for P2P systems. We systematically examine the inherent challenges and situations. Our findings show that some aspects of P2P systems can make the software testing activity difficult, such as validation of global properties and consensus, peer churn, a large volume of data, malformed data, reproducibility, and non-determinism. Several testing techniques have been used in this context, such as model-based testing, fault injection, fuzzing, stress testing, differential testing, conformance testing, and concurrency testing. We highlight fuzzing for testing malformed data, global properties, parallel events, and model-based testing as a powerful technique to abstract the complexities and varieties of P2P applications during the testing activities. Moreover, blockchain applications are the most addressed P2P application explored by the studies, where most papers were published in 2023, highlighting the theme’s relevance.

Mutation testing is a technique that assesses the effectiveness of a set of test cases by introducing changes to the source code and checking whether the test cases can detect them. However, mutation testing is costly, and many academic efforts have been directed to improve its effectiveness and reduce costs. One of the challenges related to mutation testing remains in the equivalent mutant problem. Fuzzing, as a search technique, can find test cases that the developers might not have addressed in unit testing, and it could be used to identify equivalent mutants. In this paper, we present a preliminary study that investigates the use of differential fuzzing to identify equivalent mutants. To identify equivalent mutants, one approach is to set a timeout period after which any surviving mutants are considered equivalent. In our experiment, a 3-minute timeout yielded an accuracy rate of 97%. In conclusion, differential fuzzing can be used to identify equivalent mutants accurately at a reasonable time, especially for projects that maintain a robust seed corpus for fuzzing.