JHJunxiao Han

Junxiao Han

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Junxiao Han has a total of 82 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • What Makes a Good Model Card?

    by Haochen Lu, Junxiao Han, Yujie Huang, Jiakun Liu, Lingfeng Bao, Song Han, David Lo, Shuiguang Deng

    Hugging Face (HF) is an increasingly popular platform for sharing and discovering machine learning (ML) and artificial intelligence (AI) models. Model cards document these models, and good model cards serve as a vital communication bridge between model developers and users. Without good model cards, it is challenging for users to understand the models and for developers to maintain them. Prior research focused on the documentation process and automatic generation of model cards, and identified the need for high‐quality datasets. Understanding the characteristics of good model cards can help improve the quality of model cards, which is essential for model developers and users. To address these gaps, we study the characteristics of high‐quality model cards. We collect 758 sampled models along with their model cards and adapt a prior framework to manually identify seven categories of content (i.e., categories of “What,” “Why,” “How,” “When,” “Who,” “References,” and “Other”) and their corresponding themes (e.g., “Model Description” in the “What” category) for these model cards. Then, we conduct a survey to explore what constitutes a high‐quality model card. After validation, we define a high‐quality model card as one that incorporates the “Model Overview” and “Model Evaluation” content themes from the “What” category, along with the “Model Usage Guide” content theme from the “How” category, that is, a model card that provides both a brief or detailed introduction to the model and instructions for its usage. Then, we train classifiers to investigate whether high‐quality model cards can be automatically identified. The random forest model achieves the best performance, with an ROC‐AUC of 94.57%. We also investigate the determinants of the quality of model cards. We find that the total word count of the content of model cards, the number of tables that show the key evaluation metrics and results, external links that show the references of external resources, and code blocks that are used to show examples of how to use the model are several of the most significant features that determine the quality of model cards. Finally, we explore the relationship between model card quality and model survival, and find that models with high‐quality model cards tend to remain active significantly longer than those with low‐quality model cards. Our findings provide insights into high‐quality model cards and practical suggestions for model developers, users, researchers, and the HF platform.

  • Open Access

    Better Call Grep: Evaluating and Improving Grep-Like Lexical Retrieval for Repository-Level Code Completion

    by Baoyi Wang, Xingliang Wang, Guochang Li, Chen Zhi, Junxiao Han, Xinkui Zhao, Nan Wang, Shuiguang Deng, Jianwei Yin

    Repository-level code completion remains challenging for large language models (LLMs), as it requires reasoning over cross-file dependencies while under limited context windows. To address this challenge, prior work has adopted Retrieval-Augmented Generation (RAG) frameworks based on semantic indexing or structure-aware graph analysis. Although effective, these approaches introduce substantial computational overhead for index construction and maintenance, which hinders their practicality in real-world development. Motivated by common developer workflows that rely on lightweight search utilities (e.g., ripgrep) to locate relevant code, we revisit a fundamental yet underexplored question: how far can simple, index-free lexical retrieval go in supporting repository-level code completion before more complex retrieval mechanisms become necessary? To answer this question, we systematically explore the potential of lightweight, index-free, intent-aware lexical retrieval through extensive empirical analysis. We first introduce Naive GrepRAG, a baseline framework where LLMs autonomously generate ripgrep commands to localize relevant context. Our preliminary experiments show that even this basic implementation achieves performance comparable to sophisticated graph-based baselines. Further analysis reveals that its effectiveness stems from retrieving code fragments that are lexically precise and spatially closer to the completion site. However, we identify key limitations of this approach, including sensitivity to noisy matches caused by high-frequency ambiguous keywords and context fragmentation due to rigid truncation boundaries. To address these issues, we propose GrepRAG, which augments lexical retrieval with a lightweight post-processing pipeline featuring identifier-weighted re-ranking and structure-aware deduplication. Extensive evaluation on CrossCodeEval and RepoEval_Updated demonstrates that GrepRAG consistently outperforms state-of-the-art (SOTA) methods. In particular, on CrossCodeEval, GrepRAG achieves 7.04–15.58% relative improvement in code exact match (EM) over the best baseline.

  • Open Access

    Code Refinement with Repository Context: How Far are We?

    by Ke Wang, Peng Lan, Jiakun Liu, Shengbing Ren, Lingfeng Bao, Junxiao Han, David Lo, Zhifang Liao

    Code refinement, a process of revising flawed code changes based on the code review comments, is a fundamental practice in software development for ensuring software quality and maintainability. Recent approaches using large language models (LLMs) show promise in automating this process, yet they typically rely only on review comments and a single code snippet, neglecting broader repository context. When a code refinement task requires invoking reviewer-unspecified identifiers (e.g., variables, functions, or classes) that are defined elsewhere in the repository, the LLM must accurately predict the exact identifier signature and usage, otherwise it could easily result in fatal errors. Prior work demonstrates that 90.59% of such knowledge-conflicting hallucinations cause all test cases to fail. Notably, over 60% of real-world code refinement scenarios in most popular Github repositories depend on repository-level knowledge, yet none of the existing code refinement works leverage this information. Moreover, current datasets do not distinguish between context-dependent and context-independent instances, contain many low-quality instances, and rely on lexical metrics (e.g., BLEU) that poorly correlate with human judgment. These limitations hinder reliable evaluation of code refinement tools in real-world, context-dependent scenarios. To address this gap, we introduce ReCoRe-Bench , a high-quality benchmark of 1,000 code refinement instances from 328 Python, Java, and JavaScript repositories that focused on one of the most challenging code refinement scenarios that strictly requires repository-level knowledge reasoning. To ensure repository-level context is required and to achieve strong review-revision alignment, we applied a strict identifier-based heuristic filtering pipeline and manually filtered low quality instances containing vague comments or unrelated review comments. To better evaluate whether an LLM correctly retrieves and resolves identifiers defined in the repository, instead of inferring them from limited local context, we propose a new evaluation metric, ReDIs Match. Additionally, to address the weaknesses of current LLM-powered code refinement methods identified by ReCoRe-Bench , we propose a straightforward method, RepoRefiner , which retrieves repository-level context by collecting the full file content, extracting definitions of its identifiers, and summarizing these contexts to support code refinement. The experiment results show that the RepoRefiner outperforms current code refinement baselines, and the repository-level context is essential for the reliable code refinement. Our findings underscore both the promise and current limitations of LLMs in repository-level code refinement scenarios, highlighting the need for more effective static analysis techniques and prompt engineering for broader repository-level context exploration.

  • Open Access

    Hallucinations in LLM-Based Code Summarization: Unveiling, Detection, and Mitigation

    by Guanghua Wan, Yuanning Feng, Yao Wan, Zhaoyang Chu, Zhangqian Bi, Junxiao Han, Zhou Zhao, Hongyu Zhang, Pingpeng Yuan, Xuanhua Shi, et al.

    Code summarization plays a vital role in program comprehension and software maintenance by generating natural language descriptions to summarize the semantics of code. While Large Language Models (LLMs) have shown remarkable performance in this area, recent empirical studies reveal a critical limitation: LLMs are prone to hallucinations, producing summaries that are factually inaccurate or unfaithful to the source code, potentially misleading developers. In this paper, we propose to unveil, detect, and mitigate hallucinations in LLM-based code summarization. First, we construct Hallu-Eval, a novel dataset for unveiling hallucination phenomena and rigorously evaluating the effectiveness of hallucination detection and mitigation in LLM-based code summarization. It comprises both original code snippets to capture naturally occurring hallucinations and their semantically perturbed counterparts, which are designed to systematically induce challenging logical hallucinations, all complemented with manual hallucination annotations on a curated testbed of 800 code-summary pairs. Next, we propose Hallu-Det, a synergistic approach that combines direct entity-level detection to identify explicit hallucinations with a synonymous mutation-based refinement to reliably confirm or refute more ambiguous cases. Finally, we introduce Hallu-Shield, an inference-time mitigation approach that leverages an external value model to guide LLMs toward producing more faithful summaries without costly retraining of the LLM itself. Extensive experiments show that Hallu-Eval effectively triggers hallucinations, increasing the hallucination rate of models such as Qwen2.5-Coder-7B from 17% to 97% on perturbed code. Our detection approach, Hallu-Det, achieves the best performance among baselines, reaching an F1-score of 0.95 for summaries generated by Qwen2.5-Coder-7B. Moreover, our mitigation method, Hallu-Shield, reduces hallucination rates. For example, it lowers the rate from 66% to 59%, a 10.6% relative reduction, on DeepSeek-Coder-6.7B, while simultaneously improving summary quality, achieving a 74.0% win rate evaluated by an LLM-as-a-judge majority vote ensemble.

  • Open Access

    Cloud-assisted LLM-enhanced datasets for AST hierarchy-aware code summarization model

    by Junsan Zhang, Yudie Yan, Junxiao Han, Ao Lu, Juncai Guo, Javad Pourzamani

  • Open Access

    From LLMs to Agents: A Comparative Evaluation of LLMs and LLM-based Agents in Security Patch Detection

    by Junxiao Han, Zheng Yu, Lingfeng Bao, Jiakun Liu, Yao Wan, Jianwei Yin, Shuiguang Deng, Song Han

    The widespread adoption of open-source software (OSS) has accelerated software innovation but also increased security risks due to the rapid propagation of vulnerabilities and silent patch releases. In recent years, large language models (LLMs) and LLM-based agents have demonstrated remarkable capabilities in various software engineering (SE) tasks, enabling them to effectively address software security challenges such as vulnerability detection. However, systematic evaluation of the capabilities of LLMs and LLM-based agents in security patch detection remains limited. To bridge this gap, we conduct a comprehensive evaluation of the performance of LLMs and LLM-based agents for security patch detection. Specifically, we investigate three methods: Plain LLM (a single LLM with a system prompt), Data-Aug LLM (data augmentation based on the Plain LLM), and the ReAct Agent (leveraging the thought-action-observation mechanism). We also evaluate the performance of both commercial and open-source LLMs under these methods and compare these results with those of existing baselines. Furthermore, we analyze the detection performance of these methods across various vulnerability types, and examine the impact of different prompting strategies and context window sizes on the results. Our findings reveal that the Data-Aug LLM achieves the best overall performance, whereas the ReAct Agent demonstrates the lowest false positive rate (FPR). Although baseline methods exhibit strong accuracy, their false positive rates are significantly higher. In contrast, our evaluated methods achieve comparable accuracy while substantially reducing the FPR. These findings provide valuable insights into the practical applications of LLMs and LLM-based agents in security patch detection, highlighting their advantage in maintaining robust performance while minimizing false positive rates.