Xiaokuan Zhang

Large Language Models have become powerful tools for programming. However, they often overlook essential security practices, producing code with vulnerabilities. Despite this risk, existing work offers limited guidance on steering LLMs toward secure code generation and lacks systematic analysis of how effectively LLMs repair vulnerable code. In this work, we investigate how LLMs can be guided to prevent and repair code vulnerabilities. Specifically, we examine whether self-generated vulnerability hints help models avoid insecure code, and evaluate how different feedback levels influence vulnerability repair . Our study considers proprietary and open-weight models across multiple scales and uses established benchmarks covering diverse vulnerability types. Our results show that self-generated vulnerability hints meaningfully reduce insecure code, with effectiveness depending strongly on relevance and preciseness. We further find that more directive hints, which explain how it could arise in the task and specify how to avoid it, more effectively prevent vulnerable code. For post-hoc vulnerability repair, raw detection-tool feedback improves security across all models, while detailed, actionable explanations provide further gains on two of the three benchmarks, especially for models with stronger instruction-following capabilities. Yet, this explained feedback does not consistently outperform the raw feedback for the benchmark containing real-world tasks triggering multiple weaknesses.

With more than 20,000 Common Vulnerabilities and Exposures (CVEs) reported annually, software vulnerabilities represent a critical cybersecurity challenge. This volume has intensified the demand for automated detection and analysis, motivating the integration of large language models (LLMs) for such tasks. However, existing vulnerability benchmarks are not suitable for evaluating LLMs' capabilities in vulnerability assessment, as most of them 1) rely on narrow data sources, 2) lack deep context, and 3) focus on single-turn Q&A rather than realistic, multi-stage analyst workflows. To address this gap, we introduce OmniVul, a comprehensive multi-turn benchmark for LLM-based vulnerability assessment. OmniVul comprises 2,000 CVEs with question–answer pairs spanning 23 attributes, including detection, code localization, root cause analysis, and patch suggestion. We employ an automated workflow to aggregate multi-source data via Retrieval-Augmented Generation (RAG), ensuring quality through LLM-as-a-Judge filtering and conformal prediction calibrated by human expert annotations. An evaluation of five state-of-the-art LLMs on OmniVul reveals distinct performance gaps, with top-1 accuracy remaining below 50% on average for vulnerable code detection and CVE identification. Our evaluation also demonstrates that current models lack critical reasoning capabilities for reliable vulnerability assessment. These results highlight the importance of OmniVul for advancing research in evaluating and fine-tuning LLMs for vulnerability assessment.

As the field of Web3 continues its rapid expansion, the security of Web3 authentication, often the gateway to various Web3 applications, becomes increasingly crucial. Despite its widespread use as a login method by numerous Web3 applications, the security risks of Web3 authentication have not received much attention. This paper investigates the vulnerabilities in the Web3 authentication process and proposes a new type of attack, dubbed blind message attacks. In blind message attacks, attackers trick users into blindly signing messages from target applications by exploiting users' inability to verify the source of messages, thereby achieving unauthorized access to the target application. We have developed Web3AuthChecker, a dynamic detection tool that interacts with Web3 authentication-related APIs to identify vulnerabilities. Our evaluation of real-world Web3 applications shows that a staggering 75.8% (22/29) of Web3 authentication deployments are at risk of blind message attacks. In response to this alarming situation, we implemented Web3AuthGuard on the open-source wallet MetaMask to alert users of potential attacks. Our evaluation results show that Web3AuthGuard can successfully raise alerts in 80% of the tested Web3 authentications. We have responsibly reported our findings to vulnerable websites and have been assigned two CVE IDs.