CZChengyu Zhang

Chengyu Zhang

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article

Previous Organizations / Institutes

The organizations / institutes from the scholar's other articles in the past ten years


Chengyu Zhang has a total of 44 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    An Empirical Study of False Negatives and Positives of Static Code Analyzers From the Perspective of Historical Issues

    by Han Cui, Jingjing Liang, Menglei Xie, Jiahao Peng, Ting Su, Chengyu Zhang, Shin Hwei Tan

    Static code analyzers are widely used to help find program flaws. However, in practice the effectiveness and usability of such analyzers is affected by the problems of false negatives (FNs) and false positives (FPs). This paper aims to investigate the FNs and FPs of such analyzers from a new perspective, i.e. , examining the historical issues of FNs and FPs of these analyzers reported by their maintainers, users and researchers in their issue repositories — each of these issues manifested as a FN or FP of these analyzers in the history and has already been confirmed and fixed by the analyzers’ developers. To this end, we conduct the first systematic study on a broad range of 1257 historical issues of FNs/FPs from four popular rule-based static code analyzers for Java ( i.e. , PMD , SpotBugs , SonarQube , and ErrorProne ). All these issues have been confirmed and fixed by the developers. We investigated these issues’ root causes and the characteristics of the corresponding issue-triggering programs. It reveals several new interesting findings and implications on mitigating FNs and FPs. Furthermore, guided by some findings of our study, we designed a metamorphic testing strategy to find FNs and FPs. This strategy successfully found 15 new issues of FNs/FPs, 12 of which have been confirmed and 9 have already been fixed by the developers. Our further manual investigation of the studied analyzers revealed one rule specification issue and additional three FNs/FPs due to the weaknesses of the implemented static analysis. We have made all the artifacts (datasets and tools) publicly available at https://zenodo.org/doi/10.5281/zenodo.11525129 .

  • Open Access

    Understanding and Mitigating Hallucinations in Industrial LLM-based Unit Test Generation

    by Yanlun Tu, Ziyue Zhou, Cheng Xu, Jingling Sun, Shuai Feng, Chengyu Zhang

  • Open Access

    Red Teaming LLMs via Linguistic-Aware Fuzzing

    by Shuai Yuan, Nian Luo, Jingling Sun, Yihao Huang, Chengyu Zhang

    Safety alignment aims to prevent Large Language Models (LLMs) from producing harmful content. However, safety alignment remains vulnerable to malicious instructions. Red teaming is a critical methodology for identifying such vulnerabilities in LLMs. Existing approaches often rely on jailbreak templates or rule-based transformation, limiting the diversity of generated tests and the continued testing capability of these test approaches. To address these limitations, we propose Lingfuzz, a linguistic-aware fuzzing framework for continuing red teaming LLMs. The key idea of this framework is to mutate the existing malicious instructions at the lexical and syntactic levels, while keeping the malicious intentions of the instructions. Such mutations enable generating diverse malicious instructions due to the unlimited space of lexical and syntactic choices, while having a continued testing capability by iteratively mutating the mutants. We evaluated Lingfuzz on five aligned commercial LLMs and one open-source LLM in black-box testing. The results show that Lingfuzz triggers safety alignment vulnerabilities in 71.0% of the cases in JailbreakBench benchmark, higher than the second-highest baseline of 63.8%. Lingfuzz also possesses outstanding multilingual testing capabilities that far exceed other baselines with approximately 24% ESR advantage testing in Jailbench benchmark. The malicious instructions generated by Lingfuzz have almost three times higher diversity than previous work according to the self-BLEU metric. Lingfuzz also demonstrates strong continued testing capability by showing five times less sensitivity to the LLM evolution than other approaches.


  • Open Access

    Validating Soundness and Completeness in Pattern-Match Coverage Analyzers

    by Cyril Flurin Moser, Thodoris Sotiropoulos, Chengyu Zhang, Zhendong Su

    Pattern matching is a powerful mechanism for writing safe and expressive conditional logic. Once primarily associated with functional programming, it has become a common paradigm even in non-functional languages, such as Java. Languages that support pattern matching include specific analyzers, known as pattern-match coverage analyzers, to ensure its correct and efficient use by statically verifying properties such as exhaustiveness and redundancy. However, these analyzers can suffer from soundness and completeness issues, leading to false negatives (unsafe patterns mistakenly accepted) or false positives (valid patterns incorrectly rejected). In this work, we present a systematic approach for validating soundness and completeness in pattern-match coverage analyzers. The approach consists of a novel generator for algebraic data types and pattern-matching statements, supporting features that increase the complexity of coverage analysis, such as generalized algebraic data types. To establish the test oracle without building a reference implementation from scratch, the approach generates both exhaustive and inexhaustive pattern-matching cases, either by construction or by encoding them as SMT formulas. The latter leads to a universal test oracle that cross-checks coverage analysis results against a constraint solver, exposing soundness and completeness bugs in case of inconsistencies. We implement this approach in Ikaros, which we evaluate on three major compilers: Scala, Java, and Haskell. Despite pattern-match coverage analyzers being only a small part of these compilers, Ikaros has uncovered 16 bugs, of which 12 have been fixed. Notably, 7 instances were important soundness bugs that could lead to unexpected runtime errors. Additionally, Ikaros provides a scalable framework for extending it to any language with ML-like pattern matching.


  • Open Access

    Finding Logic Bugs in Spatial Database Engines via Affine Equivalent Inputs

    by Wenjing Deng, Qiuyang Mang, Chengyu Zhang, Manuel Rigger

    Spatial Database Management Systems (SDBMSs) aim to store, manipulate, and retrieve spatial data . SDBMSs are employed in various modern applications, such as geographic information systems, computer-aided design tools, and location-based services. However, the presence of logic bugs in SDBMSs can lead to incorrect results, substantially undermining the reliability of these applications. Detecting logic bugs in SDBMSs is challenging due to the lack of ground truth for identifying incorrect results. In this paper, we propose an automated geometry-aware generator to generate high-quality SQL statements for SDBMSs and a novel concept named Affine Equivalent Inputs (AEI) to validate the results of SDBMSs. We implemented them as a tool named Spatter ( Spat ial DBMS Tes ter ) for finding logic bugs in four popular SDBMSs: PostGIS, DuckDB Spatial, MySQL, and SQL Server. Our testing campaign detected 34 previously unknown and unique bugs in these SDBMSs, of which 30 have been confirmed, and 18 have already been fixed. Our testing efforts have been well appreciated by the developers. Experimental results demonstrate that the geometry-aware generator significantly outperforms a naive random-shape generator in detecting unique bugs, and AEI can identify 14 logic bugs in SDBMSs that were totally overlooked by previous methodologies.


  • Open Access

    SMT2Test: From SMT Formulas to Effective Test Cases

    by Chengyu Zhang, Zhendong Su

    One of the primary challenges in software testing is generating high-quality test inputs and obtaining corresponding test oracles. This paper introduces a novel methodology to mitigate this challenge in testing program verifiers by employing SMT (Satisfiability Modulo Theories) formulas as a universal test case generator. The key idea is to transform SMT formulas into programs and link the satisfiability of the formulas with the safety property of the programs, allowing the satisfiability of the formulas to act as a test oracle for program verifiers. This method was implemented as a framework named SMT2Test , which enables the transformation of SMT formulas into Dafny and C programs. An intermediate representation was designed to augment the flexibility of this framework, streamlining the transformation for other programming languages and fostering modular transformation strategies. We evaluated the effectiveness of SMT2Test by finding defects in two program verifiers: the Dafny verifier and CPAchecker. Utilizing the SMT2Test framework with the SMT formulas from the SMT competition and SMT solver fuzzers, we discovered and reported a total of 14 previously unknown defects in these program verifiers that were not found by previous methods. After reporting, all of them have been confirmed, and 6 defects have been fixed. These findings show the effectiveness of our method and imply its potential application in testing other programming language infrastructures.


  • Open Access

    An Electroencephalography Study on Cognitive Load in Visual and Textual Programming

    by Sverrir Thorgeirsson, Chengyu Zhang, Theo B. Weidmann, Karl-Heinz Weidmann, Zhendong Su

    This paper presents a comparative study of Algot, a visual programming language designed to bridge the syntax-semantics gap via liveness and programming by demonstration, and the textual programming language Python. We conducted an experimental, within-subjects study with 24 undergraduate computer science students who performed recursion-based tasks in each language while their cognitive load was measured using an electroencephalogram and a validated survey instrument. The students received a brief introduction to Algot, but were all familiar with Python. The students performed significantly better when programming in Algot, but the cognitive load levels were similar according to both instruments. Our results provide evidence that within the domain that was tested, Algot can be quickly learned, and that students do not find it more cognitively demanding than working in a familiar language.


  • Open Access

    Finding and Understanding Defects in Static Analyzers by Constructing Automated Oracles

    by Weigang He, Peng Di, Mengli Ming, Chengyu Zhang, Ting Su, Shijie Li, Yulei Sui

    Static analyzers are playing crucial roles in helping find programming mistakes and security vulnerabilities. The correctness of their analysis results is crucial for the usability in practice. Otherwise, the potential defects in these analyzers (e.g., implementation errors, improper design choices) could affect the soundness (leading to false negatives) and precision (leading to false positives). However, finding the defects in off-the-shelf static analyzers is challenging because these analyzers usually lack clear and complete specifications, and the results of different analyzers may differ. To this end, this paper designs two novel types of automated oracles to find defects in static analyzers with randomly generated programs. The first oracle is constructed by using dynamic program executions and the second one leverages the inferred static analysis results. We applied these two oracles on three state-of-the-art static analyzers: Clang Static Analyzer (CSA), GCC Static Analyzer (GSA), and Pinpoint. We found 38 unique defects in these analyzers, 28 of which have been confirmed or fixed by the developers. We conducted a case study on these found defects followed by several insights and lessons learned for improving and better understanding static analyzers. We have made all the artifacts publicly available at https://github.com/Geoffrey1014/SA_Bugs for replication and benefit the community.


  • Top Cited Papers
    Open Access

    DivLog: Log Parsing with Prompt Enhanced In-Context Learning

    by Junjielong Xu, Ruichun Yang, Yintong Huo, Chengyu Zhang, Pinjia He

    Log parsing, which involves log template extraction from semi-structured logs to produce structured logs, is the first and the most critical step in automated log analysis. However, current log parsers suffer from limited effectiveness for two reasons. First, traditional data-driven log parsers solely rely on heuristics or handcrafted features designed by domain experts, which may not consistently perform well on logs from diverse systems. Second, existing supervised log parsers require model tuning, which is often limited to fixed training samples and causes sub-optimal performance across the entire log source. To address this limitation, we propose DivLog, an effective log parsing framework based on the in-context learning (ICL) ability of large language models (LLMs). Specifically, before log parsing, DivLog samples a small amount of offline logs as candidates by maximizing their diversity. Then, during log parsing, DivLog selects five appropriate labeled candidates as examples for each target log and constructs them into a prompt. By mining the semantics of examples in the prompt, DivLog generates a target log template in a training-free manner. In addition, we design a straightforward yet effective prompt format to extract the output and enhance the quality of the generated log templates. We conducted experiments on 16 widely-used public datasets. The results show that DivLog achieves (1) 98.1% Parsing Accuracy, (2) 92.1% Precision Template Accuracy, and (3) 92.9% Recall Template Accuracy on average, exhibiting state-of-the-art performance.