Simone Do Rocio Senger de Souza

Software testing through fuzzing has gained widespread adoption for discovering security vulnerabilities, yet questions remain about its effectiveness in detecting subtle behavioral faults. This paper presents an empirical evaluation investigating the intersection of fuzzing and mutation testing, specifically examining how well fuzz targets perform when evaluated through mutation analysis. We conducted a systematic study using Bitcoin Core as our subject system, analyzing 10 different fuzz targets across various modules and evaluating their ability to detect 726 generated mutants. Our methodology involved executing fuzz targets with existing seed corpora and measuring mutation scores both with and without assertion statements to understand the role of explicit oracles in fault detection. Our findings reveal that contrary to previous studies suggesting fuzzing’s limited effectiveness in mutation testing, several fuzz targets achieved high mutation scores, with two targets reaching 100% mutant detection rates. We identified three key design patterns that significantly enhance mutant detection capabilities: (1) round-trip testing approaches that verify data integrity through serialization-deserialization cycles, (2) mathematical oracles that implement exact behavioral verification through redundant calculations, and (3) metamorphic relations that validate expected relationships between inputs and outputs. Our analysis demonstrates a positive correlation between assertion density in fuzz targets and mutation scores, with assertion removal causing substantial drops in detection rates across all targets. The study contributes empirical evidence that well-designed fuzz targets can effectively detect subtle behavioral faults beyond traditional crash-based vulnerabilities. Our results suggest that incorporating explicit oracles, metamorphic properties, and roundtrip verification mechanisms into fuzz target design significantly improves their mutation testing performance. These findings have practical implications for improving fuzzing methodologies and developing more comprehensive automated testing strategies for safety-critical software systems.

The Undergraduate Committee of the São Carlos School of Engineering (EESC-USP) has created two optative tutorial classes to guide and welcome new students from the 10 undergraduate engineering programs offered. With practical activities and motivational and informative lectures, the objective is to bring to students the academic, sociocultural and scientific dimensions that the university can offer during their studies along with opportunities to be engaged in extracurricular projects. Academic Tutoring I is provided to new students in the first semester, starting in 2022. Academic Tutoring II is offered in the second semester, starting in 2023, and was created to continue supporting first-year students at EESC. This article presents the proposal for academic tutoring, considering the main performance issues that students currently face, such as the increase in anxiety and depression among young people and the problems arising from the isolation caused by Covid-19. The results of new students' acceptance of the tutorial classes were obtained through qualitative and quantitative research and also student reports. It was evident that the subjects provided a better welcome and encouraged discussions about competencies and skills in the context of diversity and inclusion.

Over the past decade, there has been a significant surge in interest regarding the application of machine learning (ML) across various tasks. Due to this interest, the adoption of ML-based systems has gone mainstream. It turns out that it is imperative to conduct thorough software testing on these systems to ensure that they behave as expected. However, ML-based systems present unique challenges for software testers who are striving to enhance the quality and reliability of these solutions. To cope with these testing challenges, we propose novel test adequacy criteria centered on decision tree models. Our criteria diverge from the conventional method of manually collecting and labeling data. Instead, our criteria relies on the inherent structure of decision tree models to inform the selection of test inputs. Specifically, we introduce decision tree coverage (DTC) and boundary value analysis (BVA) as approaches to systematically guide the creation of effective test data that exercises key structural elements of a given decision tree model. Additionally, we also propose a mutation based criterion to support the validation of ML-based systems. Essentially, this approach involves applying mutation analysis to the decision tree structure. The resulting mutated trees are then used as a reference for selecting test data that can effectively identify incorrect classifications in ML models. To evaluate these criteria, we carried out an experiment using 16 datasets. We measured the effectiveness of test inputs in terms of the difference in model’s behavior between the test input and the training data. According to the results of the experiment, our criteria can be used to improve the test data selection for ML applications by guiding the generation of diversified test data that negatively impact the prediction performance of models.

Peer-to-peer (P2P) systems, characterized by their decentralized architecture, have become crucial to various applications such as file sharing, content distribution, and blockchain technologies. Due to their dynamic and heterogeneous nature, ensuring their reliability and robustness presents challenges. Such an issue has motivated researchers to develop testing techniques for these systems. Despite advances in research about software testing for distributed systems, there is a lack of data about their applicability and which features from P2P systems are highlighted by the proposed testing approaches. This paper provides a comprehensive overview of the state-of-the-art software testing for P2P systems. We systematically examine the inherent challenges and situations. Our findings show that some aspects of P2P systems can make the software testing activity difficult, such as validation of global properties and consensus, peer churn, a large volume of data, malformed data, reproducibility, and non-determinism. Several testing techniques have been used in this context, such as model-based testing, fault injection, fuzzing, stress testing, differential testing, conformance testing, and concurrency testing. We highlight fuzzing for testing malformed data, global properties, parallel events, and model-based testing as a powerful technique to abstract the complexities and varieties of P2P applications during the testing activities. Moreover, blockchain applications are the most addressed P2P application explored by the studies, where most papers were published in 2023, highlighting the theme’s relevance.

Mutation testing is a technique that assesses the effectiveness of a set of test cases by introducing changes to the source code and checking whether the test cases can detect them. However, mutation testing is costly, and many academic efforts have been directed to improve its effectiveness and reduce costs. One of the challenges related to mutation testing remains in the equivalent mutant problem. Fuzzing, as a search technique, can find test cases that the developers might not have addressed in unit testing, and it could be used to identify equivalent mutants. In this paper, we present a preliminary study that investigates the use of differential fuzzing to identify equivalent mutants. To identify equivalent mutants, one approach is to set a timeout period after which any surviving mutants are considered equivalent. In our experiment, a 3-minute timeout yielded an accuracy rate of 97%. In conclusion, differential fuzzing can be used to identify equivalent mutants accurately at a reasonable time, especially for projects that maintain a robust seed corpus for fuzzing.

The Android system has seen considerable growth in its vulnerability landscape due to an extensive application catalog catering to many user needs, many of which are security sensitive. This growth leads to an ever-increasing concern about security robustness; hence, security testing Android apps has gained substantial prominence in recent years. Many security professionals and tools specialize in security testing Android applications, but the quality of testing procedures varies significantly. In this paper, we present a preliminary study exploring the use of Mutation Testing to support Android security testing. We propose novel mutation operators, implement them in code, and conduct an experiment to evaluate their resemblance to real-world vulnerabilities.We test our mutants using a well-known open-source tool named mobsfscan. Our results indicate the adequacy of our operators for supporting security testing. Moreover, we reveal a potential design flaw in mobsfscan.

There has been a growing interest in machine learning due to its potential to address a myriad of problems that would otherwise be difficult to solve. Consequently, the adoption of machine learning based programs has become mainstream. Owing to this widespread adoption, it is imperative to develop automated approaches to assess the quality of machine learning-based solutions. Although significant research has been devoted to creating automated test input generation methods for machine learning programs, some promising approaches to test data generation have received limited attention. This paper introduces a property-driven approach to test data generation that leverages the training of an interpretable model, specifically a decision tree, to predict the behavior of the model under test. The tree-like structure of the resulting interpretable model provides valuable insights into the model’s behavior under test. These insights are then transformed into executable properties, enabling the generation of test data. A primary advantage of property-based testing is its capacity to generate a vast number of inputs from a single property, thereby offering a more rigorous evaluation of machine learning models. The results of our experiment suggest that our property-driven approach has the potential to generate test data that more thoroughly examine models compared to more widely used methods for evaluating the performance and generalizability of machine learning models.