Lin Chen

Background . Code coverage and mutation score serve as pivotal test effectiveness metrics used to assess a test suite’s ability to uncover actual defects. However, prior research has produced inconsistent or even conflicting findings regarding their correlation with defect detection capability, particularly concerning the impact of test suite size. Problem. The extent of the potentially confounding effect of test suite size in test effectiveness evaluation context is not clear, nor is the method to remove the potentially confounding effect, or the influence of this removal on the performance of test suite optimization. Objective . Our goal is to deeply understand how test suite size affects the true relationship between test effectiveness metrics and a test suite’s ability to detect actual defects. Method. We first employ statistical methods to examine the extent of the potentially confounding effect of test suite size in the context of test effectiveness evaluation. After that, we propose a linear regression-based method to remove the potentially confounding effect of test suite size. Finally, we empirically explore the impact of this removal method on test suite optimization. Result. Our experimental results, based on the Defects4J defect dataset, uncovers that: (1) the confounding effect of test suite size on the associations between test effectiveness metrics and defect detection capability in general exists; (2) the proposed linear regression-based method can effectively remove the confounding effect; and (3) after removing the confounding effect, mutation score demonstrates superior effectiveness in predicting test suite effectiveness, while statement coverage is the least effective metric. Furthermore, both coverage-based and mutation-based test suite reduction exhibit enhanced cost-effectiveness in defect detection, and there is a marginal improvement in the speed of defect detection for coverage-based test case prioritization. Conclusion . When using test effectiveness metrics to assess test suite effectiveness, it is crucial to eliminate the influence of test suite size.

Machine Learning (ML) software employs statistical algorithms to perform high-stake tasks in our daily lives, whose results are usually discriminatory due to protected features (e.g., gender), i.e., one part (called privileged, e.g., male) may be more likely to obtain beneficial decisions than the other part (called unprivileged, e.g., female). In alleviating the unfairness, developers have obtained widely held beliefs about the tradeoff between performance and fairness for ML software. Surprisingly, recent research on feature engineering suggests that enlarging the feature set is the perfect way to kill two birds with one stone, i.e., achieving both higher performance and fairness. However, the experiments used in the prior study did not remove the effect of protected features, which have been suggested to be excluded in both industrial applications and academic studies. As a result, the study did not fully explore the tradeoff between performance and fairness. In this article, we first conduct an empirical study to replicate this prior study after excluding the protected features and observe that there is still a tradeoff between performance and fairness with enlarging the features, i.e., more features are not perfect, which would lead to higher performance and lower fairness. Due to more features causing more collection and pre-processing budgets, we aim to search for an effective alternative. Inspired by the “less is more” principle, we propose a novel feature ranking method, Hybrid-importance and Early-validation based Feature Ranking (HEFR) , to find an efficient subset to replace the full feature set with comparable performance and fairness. Our method, HEFR, employs hybrid feature importances to combine performance and fairness and conducts early validation to check the effectiveness of hybrid importances. We conduct experiments on seven datasets and three classifiers to evaluate our method with five baselines. The results have shown that (a) HEFR is efficient for ML software feature engineering: applying HEFR to choose about 10% of features would construct ML software with better or comparable performance and fairness, and (b) HEFR is actionable with small dataset sizes: applying HEFR with only 10% data size would still help choose the proper feature subset.

Existing fine-grained predictive mutation testing studies predominantly rely on deep learning, which faces two critical limitations in practice: (1) Exorbitant computational costs. The deep learning models adopted in these studies demand significant computational resources for training and inference acceleration. This introduces high costs and undermines the cost-reduction goal of predictive mutation testing. (2) Constrained applicability. Although modern mutation testing tools generate mutants both inside and outside methods, current fine-grained predictive mutation testing approaches handle only inside-method mutants. As a result, they cannot predict outside-method mutants, limiting their applicability in real-world scenarios. We propose WITNESS, a new fine-grained predictive mutation testing approach. WITNESS adopts a twofold design: (1) With collected features from both inside-method and outside-method mutants, WITNESS is suitable for all generated mutants. (2) Instead of using computationally expensive deep learning, WITNESS employs lightweight classical machine learning models for training and prediction. This makes it more cost-effective and enabling straightforward explanations of the decision-making processes behind the adopted models. Evaluations on Defects4J projects show that WITNESS consistently achieves state-of-the-art predictive performance across different scenarios. Additionally, WITNESS significantly enhances the efficiency of kill matrix prediction. Post-hoc analysis reveals that features incorporating information from before and after the mutation are the most important among those used in WITNESS. Test case prioritization based on the predicted kill matrix shows that WITNESS delivers results much closer to those obtained by using the actual kill matrix, outperforming baseline approaches.

In the rapidly advancing field of software development, the demand for practical code translation tools has surged, driven by the need for interoperability across different programming environments. Existing learning-based approaches often need help with low-resource programming languages that lack sufficient parallel code corpora for training. To address these limitations, we propose a novel training framework that begins with monolingual seed corpora, generating parallel datasets via back-translation and incorporating compiler feedback to optimize the translation model. As a case study, we apply our method to train a code translation model for a new-born low-resource programming language, Cangjie. We also construct a parallel test dataset for Java-to-Cangjie translation and test cases to evaluate the effectiveness of our approach. Experimental results demonstrate that compiler feedback greatly enhances syntactical correctness, semantic accuracy, and test pass rates of the translated Cangjie code. These findings highlight the potential of our method to support code translation in low-resource settings, expanding the capabilities of learning-based models for programming languages with limited data availability.