Shengnan Wu

Code translation, as a challenging and fundamental task, is increasingly relying on large language models (LLMs). However, LLMs often give seemingly plausible but fallacious translations, misleading and even deceptive to debugging developers. We propose tHinter, an automated approach that frames translation error localization as a differential testing task. By fuzzing and a novel Bayesian-inspired algorithm that integrates developer heuristics into SBFL, tHinter thoroughly exposes translation errors and effectively pinpoints functionality-twisting errors. We evaluate tHinter in a Python to C++ scenario, a representative and challenging task for migrating high-level logic to performance-critical systems. Experiments show that tHinter achieves the highest F1 in localizing translation errors, outperforming all baselines. When debugging LLM-translated code with human efforts, it averagely reduces 71% lines developers need to review. Furthermore, we provide a comprehensive evaluation reference beyond traditional fault localization metrics. We integrate mixed-factorial user studies, expert validation, and SWOT-based strategic analysis to assess the perceived helpfulness and resilience within the rapidly evolving LLM landscape, offering a methodologically rigorous example for researchers in the context of AI-augmented software engineering.

Large Language Models (LLMs) hold great promise in the task of code translation. However, the lack of explainability complicates the identification of the inevitable translation errors. In this paper, we propose tHinter, a debugging tool to locate translation errors in auto-translated code. The core idea of tHinter is that correctly translated, the source and translated code should present the same functionalities, giving the same output for the same input. Hence, lines in the translated code responsible for output differences are possibly translation errors. First, tHinter employs fuzzing to generate diverse test cases that thoroughly explore the translated code. Then, tHinter relies on a heuristic algorithm to pinpoint translation errors from coverage information and differential testing execution results of those test cases. This heuristic algorithm is designed to leverage both the statistics and the expertise of developers. Comprehensive experiments with real code show its effectiveness. It reduces 71% lines developers need to review during debugging and increases the likelihood of the LLM fixing translation errors in a single query by 59%. Developers generally consider it satisfactory and helpful.

Search components in e-commerce apps, often complex AI-based systems, are prone to bugs that can lead to missed recalls—situations where items that should be listed in search results aren't. This can frustrate shop owners and harm the app's profitability. However, testing for missed recalls is challenging due to difficulties in generating user-aligned test cases and the absence of oracles. In this paper, we introduce mrDetector, the first automatic testing approach specifically for missed recalls. To tackle the test case generation challenge, we use findings from how users construct queries during searching to create a CoT prompt to generate user-aligned queries by LLM. In addition, we learn from users who create multiple queries for one shop and compare search results, and provide a test oracle through a metamorphic relation. Extensive experiments using open access data demonstrate that mrDetector outperforms all baselines with the lowest false positive ratio. Experiments with real industrial data show that mrDetector discovers over one hundred missed recalls with only 17 false positives.