Tsong Yueh Chen

Browser autofill is a browser feature that automatically fills web-form fields using stored user information. It uses semantic information from web forms to determine what to insert into individual fields. This information can include user-facing information (such as labels and placeholders) and browser-facing metadata (such as field attributes and HTML autocomplete attributes). We refer to these sources of information as semantic signals, some of which may be missing or conflicting, potentially causing browser behavior to diverge from the information conveyed to users. This motivated us to investigate how such semantic inconsistencies affect browser autofill, and whether or not we can systematically detect and mitigate their impact. We conducted an empirical study of the semantic consistency of Google Chrome’s autofill, analyzing 26,663 real-world pages containing forms (with 48,719 deduplicated forms and 159,235 fields). We constructed a human-annotated benchmark in which we determined field purpose using only user-facing information and compared it with observed Chrome autofill behavior. In this 719-field benchmark, Chrome autofilled 68.15% of the fields, with 99.18% of the fills being semantically aligned with the field purpose indicated by user-facing information. This low coverage primarily reflected missing autofill, rather than wrong-category autofill. Based on this, we introduce a metamorphic testing (MT) method that preserves the user-facing field purpose while modifying browser-facing semantic signals. We evaluated two types of transformations: Removing individual semantic signals, and introducing conflicting autocomplete metadata. Removing an individual semantic signal altered autofill behavior in 44.03% of tested mutations, always by reducing coverage rather than producing wrong-category autofill. In contrast, Chrome’s autofill category matched the category specified by the conflicting autocomplete metadata in 196 of 198 conflicting-autocomplete mutations (98.99%), resulting in wrong-category autofill. Based on these observations, we developed a consistency checker that uses a large language model (LLM). The checker infers field purpose from user-facing information and compares it with browser-facing metadata to detect semantic conflicts, achieving an F1 score of 98.46% on an 850-case conflict-detection benchmark. We then evaluated mitigation on the 196 confirmed wrong-category autofill cases, eliminating wrong-category autofill in 192 cases. Overall, removing individual semantic signals tended to suppress autofill, whereas conflicting autocomplete metadata reliably produced wrong-category autofill.

Simulation-based testing of autonomous driving systems (ADS) must uncover realistic and diverse failures in dense, heterogeneous traffic. However, existing search-based seeding methods (e.g., genetic algorithms) struggle in high-dimensional spaces, often collapsing to limited modes and missing many failure scenarios. We present PtoP, a framework that combines adaptive random seed generation with Stein Variational Gradient Descent (SVGD) to produce diverse, failure-inducing initial conditions. SVGD balances attraction toward high-risk regions and repulsion among particles, yielding risk-seeking yet well-distributed seeds across multiple failure modes. PtoP is plug-and-play and enhances existing online testing methods (e.g., reinforcement learning--based testers) by providing principled seeds. Evaluation in CARLA on two industry-grade ADS (Apollo, Autoware) and a native end-to-end system shows that PtoP improves safety violation rate (up to 27.68%), scenario diversity (9.6%), and map coverage (16.78%) over baselines.

With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in single-turn testing scenarios. However, multi-turn interaction is the common realworld usage of dialogue systems, yet testing methods for such interactions remain underexplored. This is largely due to the oracle problem in multi-turn testing, which continues to pose a significant challenge for dialogue system developers and researchers. In this paper, we propose MORTAR, a metamorphic multi-turn dialogue testing approach, which mitigates the test oracle problem in testing LLM-based dialogue systems. MORTAR formalises multiturn testing for dialogue systems, and automates the generation of question-answer dialogue test cases with multiple dialogue-level perturbations and metamorphic relations (MRs). The automated MR matching mechanism provides MORTAR more flexibility and efficiency in testing. When applied to eight LLM-based dialogue systems as test objects, MORTAR reveals 51% more bugs per test case than the best-performing single-turn metamorphic testing baseline, with 24.6% higher precision and over 42% of detected bugs being unique.

Metamorphic Testing (MT) is a powerful technique to alleviate the test oracle problem of DNN testing. At its core is a set of Metamorphic Relations (MRs), which indicate necessary properties that multiple test inputs and their outputs should satisfy. Recently, researchers have proposed to compose multiple individual MRs to create new Composite Metamorphic Relations (CMRs), in the hope of enhancing the cost-effectiveness of MT. However, the potential benefit of such compositions is yet to be systematically explored for DNN based systems. In this paper, we present the first empirical study that investigates the relative test effectiveness of CMRs over individual MRs in DNN testing. Our experiment is performed under five popular image recognition models and seven representative component MRs, based on which a total of 3,612 CMRs and more than 200 million pairs of source and follow-up test images are generated and exercised. The experimental results highlight the advantages of employing CMRs in DNN testing, as they generally exhibit superior failure and fault revelation effectiveness compared to any individual MR. Moreover, to further illuminate the conditions that allow CMRs to maximise their performance in DNN testing, we propose to analyse the geometric relationships of MRs in the latent embedding space of DNNs, thereby quantifying the extent to which different MRs complement each other. Accordingly, by opting for highly complementary MRs in composition, we can create CMRs that are most likely to enhance test effectiveness.

Metamorphic testing has become one mainstream technique to address the notorious oracle problem in software testing, thanks to its great successes in revealing real-life bugs in a wide variety of software systems. Metamorphic relations, the core component of metamorphic testing, have continuously attracted research interests from both academia and industry. In the last decade, a rapidly increasing number of studies have been conducted to systematically generate metamorphic relations from various sources and for different application domains. In this article, based on the systematic review on the state of the art for metamorphic relations’ generation, we summarize and highlight visions for further advancing the theory and techniques for identifying and constructing metamorphic relations and discuss promising research directions in related areas.