JZJuan Zhai

Juan Zhai

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Juan Zhai has a total of 80 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    CIPIHunter: Detecting Configuration-Induced Prediction Instability in Deep Learning Frameworks

    by Yanzhou Mu, Shuo Meng, Mijung Kim, Xiang Chen, Chunrong Fang, Zhenyu Chen, Juan Zhai

    Deep learning (DL) frameworks support diverse execution configurations, including precision modes and model optimizations. Even with a fixed model, input, and framework version, changing these configurations can change model predictions, causing Configuration-Induced Prediction Instability (CIPI). It arises when configuration-dependent numerical perturbations propagate and cause potentially harmful model-level behavior. Existing API-level and model-level testing approaches are not well suited to detecting CIPI, as these approaches lack configuration-conditioned exploration and propagation-aware diagnosis. Therefore, we present CIPIHunter, which combines controlled configuration-pair testing, sensitivity-oriented mutation, feedback-guided search, and intermediate tracing to localize suspicious operators or operator groups. On 13 PyTorch models under a 100-round budget, CIPIHunter reports the highest raw CIPI count and 22 deduplicated Type I manifestations, 83.3% more than the strongest baseline, while remaining competitive in overall deduplicated manifestation coverage. In total, we submit 29 PyTorch issue reports. Developers have acknowledged eight, rejected five, and marked four as duplicates; nine remain pending, and three have been closed without an explicit determination. Separately, three reports are labeled “high-priority”. These results demonstrate the practical utility of CIPI testing for framework quality assurance.

  • Open Access

    From Draft to Precision: Iterative Agentic Framework for Intent-Aware Code Summarization

    by Yifei Ge, Chunrong Fang, Zhenyu Chen, Juan Zhai

    Code summarization aims to generate natural language (NL) descriptions for code snippets to assist developers in understanding and maintaining programs. Recent advances in large language models (LLMs) have substantially improved the quality of automatic code summarization. In real-world projects, code comments are written to serve different developer intents, which has motivated the development of intent-aware summarization methods. However, existing intent-aware methods still struggle to fully capture developer intent, often producing incomplete or misaligned summaries. Our empirical study reveals that real-world comments are typically refined through multiple rounds of developer revision, which we believe is key to achieving high quality. However, current methods typically overlook this iterative refinement. To bridge this gap, we propose a plan-conditioned revision framework that improves code summarization through iterative revision. It integrates two collaborating agents: a Generator that produces and revises summaries, and a Reviewer that assesses the generated summary and drafts targeted revision plans to guide subsequent edits using available contextual information. By iteratively refining summaries with revision plans and contextual information, our framework yields summaries that are more accurate, complete, and better aligned with developer intent. Extensive experiments on an intent-annotated CSN-Java benchmark demonstrate the effectiveness of our method. On automatic metrics, our method achieves a 28.75% improvement over the state-of-the-art baseline methods across all intents. Besides, in human studies, our framework yields average gains of over 20% in usefulness, adequacy, and intent alignment. It also achieves the highest Top-1 preference (over 40% across all intents), underscoring its practical advantage for developers

  • Open Access

    Deep Learning Framework Testing via Heuristic Guidance Based on Multiple Model Measurements

    by Yinglong Zou, Juan Zhai, Chunrong Fang, Yanzhou Mu, Jiawei Liu, Zhenyu Chen

    Deep learning frameworks serve as the foundation for developing and deploying deep learning applications. To enhance the quality of deep learning frameworks, researchers have proposed numerous testing methods using deep learning models as test inputs. However, existing methods predominantly measure model bug detection effectiveness as heuristic indicators, presenting three critical limitations. Firstly, existing methods fail to quantitatively measure model's operator combination variety, potentially missing critical operator combinations that could trigger framework bugs. Secondly, existing methods neglect measuring and heuristically guiding the model execution time, resulting in the omission of numerous models potential for detecting more framework bugs within limited testing time. Thirdly, existing methods overlook correlation between different model measurements, relying simply on single-indicator heuristic guidance without considering their trade-offs. To overcome these limitations, we propose DLMMM, the first deep learning framework testing method to include multiple model measurements into heuristic guidance and fuse these measurements to achieve their trade-offs. DLMMM firstly quantitatively measures model's bug detection performance, operator combination variety, and model execution time. After that, DLMMM fuses these measurements based on their correlation to achieve their trade-offs. To further enhance testing effectiveness, DLMMM designs multi-level heuristic guidance for test input model generation. We apply DLMMM to test three widely used deep learning frameworks (including TensorFlow, PyTorch, and MindSpore). The experimental results show that DLMMM outperforms state-of-the-art methods in effectiveness and efficiency. The test input models generated by DLMMM achieve a 250.13% improvement in operator combination variety. The average model execution time of DLMMM is within 35% of all state-of-the-art methods. DLMMM detects 19 new crashes and 175 new NaN & inconsistency bugs. Among detected crashes, 16 have been confirmed by developers and two have been fixed in the following version.

  • Open Access

    Bugs in Modern LLM Agent Frameworks: An Empirical Study

    by Xinxue Zhu, Jiacong Wu, Xiaoyu Zhang, Tianlin Li, Yanzhou Mu, Juan Zhai, Chao Shen, Chunrong Fang, Yang Liu


  • Open Access

    Understanding LLM-Centric Challenges for Deep Learning Frameworks: An Empirical Analysis

    by Yanzhou Mu, Rong Wang, Juan Zhai, Chunrong Fang, Xiang Chen, Jiacong Wu, An Guo, Jiawei Shen, Bingzhuo Li, Zhenyu Chen

    Large language models (LLMs) have achieved remarkable success across diverse applications, but their development and deployment depend heavily on deep learning (DL) frameworks. The massive scale, long execution cycles, and complex workflows of LLMs place stringent demands on framework usability, functionality, and reliability. When these demands are not well supported, developers may suffer from reduced efficiency, severe failures, and substantial resource waste. However, a fundamental question remains underexplored: what challenges do DL frameworks face in supporting LLMs? To answer this question, we analyze issue reports from three major DL frameworks (i.e., MindSpore, PyTorch, and TensorFlow) and eight associated LLM toolkits such as DeepSpeed. Based on a manual review of these reports, we construct a taxonomy that captures LLM-centric framework bugs, user requirements, and user questions. We then refine and enrich this taxonomy through interviews with 11 LLM users and eight DL framework developers. Using the taxonomy and interview findings, our study further reveals key technical challenges and mismatches between LLM user needs and developer priorities. Overall, our contributions are threefold: (1) we develop a comprehensive taxonomy comprising five question themes (11 sub-themes), five requirement themes (17 sub-themes), and ten bug themes (47 sub-themes); (2) we assess the importance and priority of different categories in the taxonomy based on insights from LLM users and framework developers; and (3) we identify five key findings across the LLM development lifecycle and propose five actionable recommendations to improve the quality of DL frameworks in supporting LLMs. Our results highlight critical limitations in current DL frameworks and offer concrete guidance for advancing their support for future LLM development, deployment, and applications.

  • Open Access

    Citadel : Context Similarity Based Deep Learning Framework Bug Finding

    by Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Shiwei Wang, Chao Shen

    With the application of Deep Learning (DL) technology, tools of DL framework testing are in high demand. Existing DL framework testing tools have limited coverage of bug types. For example, they lack the capability of effectively finding performance bugs, which are critical for DL models regarding performance, economics, and the environment. Moreover, existing tools are inefficient, generating hundreds of test cases with few trigger bugs. In this article, we propose Citadel , a method that accelerates bug finding in terms of efficiency and effectiveness. We observe that many DL framework bugs are similar due to the similarity of operators and algorithms belonging to the same family. Orthogonal to existing bug-finding tools, Citadel aims to find new bugs that are similar to reported ones that have known test oracles. Citadel defines context similarity to measure the similarity of DL framework API pairs and automatically generates test cases with oracles for APIs that are similar to the problematic APIs in existing bug reports. Citadel effectively detects 58 and 66 API bugs on PyTorch and TensorFlow (excluding those rejected by developers or duplicates of prior reports), many of which, e.g., 13 performance bugs, cannot be detected by existing tools. Moreover, 35.40% of test cases generated by Citadel can trigger bugs significantly transcending the state-of-the-art method (3.90%).


  • Open Access

    Detecting and Analyzing Fine-grained Third-party Library Dependencies in Solidity Smart Contracts

    by Sicheng Hao, Yuhong Nan, Zeqin Liao, Juan Zhai, Zibin Zheng

    Solidity is the primary programming language for writing smart contracts. As a lightweight language, Solidity does not have a unified way to manage third-party library (TPL) dependencies. Instead, the copy-and-paste pattern and other dependency managers such as NPM and Git submodules have become alternatives. However, these mechanisms significantly increase the complexity of TPL usage with security concerns. Similar to other programming language ecosystems, incorrect TPL usage can influence the reliability of contracts and even introduce vulnerabilities from outdated versions. Therefore, there is an urgent need to understand and comprehend Solidity TPL dependencies. In this work, we conduct a comprehensive study on TPL dependency usage in Solidity. To achieve this, we first present SPADE , which leverages additional metadata (e.g., package and remapping configurations) to infer fine-grained TPL dependencies with version and contract details in various Solidity projects. With SPADE , we investigate a broad spectrum of 5,242 Solidity repositories to understand the TPL dependencies, including their landscape and version-level usage. Our research reveals a set of interesting and important findings that can be beneficial to the Solidity ecosystem. In particular, TPL dependencies are prevalent in Solidity, but the version management remains inadequate. The propagation of vulnerability is severe, affecting 8.87% of the repositories. Finally, we use on-chain contracts to validate the findings and provide suggestions for future research and development on Solidity TPL dependencies.

  • Open Access

    Scalpel: Automotive Deep Learning Framework Testing via Assembling Model Components

    by Yinglong Zou, Juan Zhai, Chunrong Fang, An Guo, Jiawei Liu, Zhenyu Chen

    Deep learning (DL) plays a key role in autonomous driving systems. DL models support perception modules, equipped with tasks such as object detection and sensor fusion. These DL models enable vehicles to process multi-sensor inputs to understand complex surroundings. Deploying DL models in autonomous driving systems faces stringent challenges, including real-time processing, limited computational resources, and strict power constraints. To address these challenges, automotive DL frameworks (e.g., PaddleInference) have emerged to optimize inference efficiency. However, these frameworks encounter unique quality issues due to their more complex deployment environments, such as crashes stemming from limited scheduled memory and incorrect memory allocation. Unfortunately, existing DL framework testing methods fail to detect these quality issues due to the failure in deploying generated test input models, as these models lack three essential capabilities: (1) multi-input/output tensor processing, (2) multi-modal data processing, and (3) multi-level data feature extraction. These capabilities necessitate specialized model components, which existing testing methods neglect during model generation. To bridge this gap, we propose Scalpel, an automotive DL frameworks testing method that generates test input models at the model component level. Scalpel generates models by assembling model components (heads, necks, backbones) to support capabilities required by autonomous driving systems. Specifically, Scalpel maintains and updates a repository of model components, generating test inputs by selecting, mutating, and assembling them. Successfully generated models are added back to enrich the repository. Newly generated models are then deployed within the autonomous driving system to test automotive DL frameworks via differential testing. The experimental results demonstrate that Scalpel outperforms existing methods in both effectiveness and efficiency. In Apollo, Scalpel detects 16 crashes and 21 NaN & inconsistency bugs. All detected bugs have been reported to open-source communities, with 10 crashes confirmed. Scalpel achieves 27.44 × and 8.5 × improvements in model generation efficiency and bug detection efficiency. Additionally, Scalpel detects nine crashes and 16 NaN & inconsistency bugs in Autoware, which shows its excellent generalization.

  • Deep Learning Framework Testing via Model Mutation: How Far Are We?

    by Yanzhou Mu, Rong Wang, Juan Zhai, Chunrong Fang, Xiang Chen, Peiran Yang, Zhixiang Cao, Ruixiang Qian, Shaoyu Yang, Zhenyu Chen

    Deep Learning (DL) frameworks are fundamental components of DL systems in their development, deployment, and execution, while defects in DL frameworks can cause severe consequences. Ensuring the quality of DL frameworks has therefore become a pressing challenge. Among the various testing techniques, model mutation has emerged as a widely adopted approach. Such methods generate mutants by applying mutation operators to DL models (e.g., structural changes or parameter edits) and then analyzing inconsistencies, crashes, or abnormal behaviors across different frameworks or hardware. Despite its effectiveness, existing methods suffer from the following limitations. First, they mainly reuse operators designed for model testing, raising doubts about their ability to expose framework-level defects. Besides, they insufficiently consider mutation constraints, such as mutation type, position, and order, which directly affect the defect detection ability of generated mutants. Finally, they rely on the limited detection range and narrow test oracles, focusing on functional correctness in model inference while overlooking defects in efficiency, resource usage, and other defects that developers care about in other stages, such as model training or deployment. These limitations result in a weak alignment with the critical defects that developers are most concerned about in practice. Motivated by these observations, this study conducts a comprehensive investigation into the effectiveness of existing mutation-based testing methods. We first collect and classify defect reports from PyTorch and MindSpore according to developers’ priority tags, building a taxonomy of seven categories and 19 sub-categories of HP defects. We then map the defects reported by five state-of-the-art methods into this taxonomy to evaluate their detection abilities. To explain these limitations, we further analyze how three key factors, mutation type, mutation position, and mutation order, affect the generated mutants. Based on the experiment results, we summarize ten findings ranging from revealing the priority of developers on fixing framework defects, evaluating the defect detection ability of existing methods, to how mutation factors affect the generated mutants. Furthermore, we reveal four limitations and their root causes of existing methods and propose four targeted optimization strategies. We further apply these strategies to COMET and successfully uncover six new defects spanning four types, including two previously unreported categories. Overall, our study identifies 38 unique framework defects, of which 30 are confirmed by developers and 12 have been fixed, demonstrating the practical value of our findings.

  • Open Access

    Breaking the Myth: Can Small Models Infer Postconditions Too?

    by Gehao Zhang, Zhenting Wang, Juan Zhai

    Formal specifications are essential for ensuring software correctness, yet manually writing them is tedious and error-prone. Large Language Models (LLMs) have shown promise in generating such specifications from natural language intents, but the giant model size and high computational demands raise a fundamental question: Do we really need large models for this task? In this paper, we show that a small, fine-tuned language model can achieve high-quality postcondition generation with much lower computational costs. We construct a specialized dataset of prompts, reasoning logs, and postconditions, then supervise the fine-tuning of a $7$B-parameter code model. Our approach tackles real-world repository dependencies and preserves pre-state information, allowing for expressive and accurate specifications. We evaluate the model on a benchmark of real-world Java bugs (Defects4J) and compare against both proprietary giants (e.g., GPT-4o) and open-source large models. Empirical results demonstrate that our compact model matches or outperforms significantly larger counterparts in syntax correctness, semantic correctness, and bug-distinguishing capability. These findings highlight that targeted fine-tuning on a modest dataset can enable small models to achieve results formerly seen only in massive, resource-heavy LLMs, offering a practical and efficient path for the real-world adoption of automated specification generation.