Zongjie Li

Coding agents powered by large language models are becoming central modules of modern IDEs. They help users to perform various complex coding tasks by invoking tools. Although powerful, tool-invocation operation in coding agents opens a substantial attack surface for adversaries. Prior work has demonstrated attacks against both general-purpose LLM agents and domain-specific agents. However, to our knowledge, no previous works focus on the security risks of tool-invocation in coding agents. To fill this gap, we conduct the first systematic, in-depth red-teaming from a tool-invocation perspective in six popular real-world coding agents: Cursor, Claude Code, Copilot, Windsurf, Cline, and Trae. Our red-teaming proceeds in two phases. In Phase 1, we conduct a prompt leakage as reconnaissance to recover system prompts and related context. Specifically, we identify a mode gap between chat generation and tool-call argument generation: during schema-driven argument completion, the model behaves as if it is performing benign structured filling and may copy hidden agent context into tool-call arguments. We instantiate this gap as ToolLeak, which exfiltrates agent-internal prompts (e.g., system prompts and tool metadata) via required tool parameters. In Phase 2, we hijack the tool-invocation behavior of the coding agent with a novel two-channel prompt injection in the tool description and the tool return. Our hijacking achieves remote code execution (RCE) on major real-world coding agents. We adaptively construct the malicious payload using leaked security information in Phase 1. Our evaluation shows that ToolLeak substantially outperforms strong prompt-leak baselines in both emulated and real-world settings. In the emulated setting, ToolLeak achieves the best overall prompt-exfiltration performance across all six simulated coding agents. On real-world coding agents, ToolLeak achieves the best pseudo-recall on 18 of 25 evaluated agent-LLM pairs. Furthermore, our red-teaming successfully hijacks all six real-world coding agents for RCE and consistently yields higher attack success rates than baseline attacks. Lastly, we present two case studies on Cursor and Claude Code to demonstrate the real-world impact of our red-teaming.

Decompiling stripped binaries to assist human reverse engineers remains a critical yet highly challenging task in software security. Prior work has developed various neural networks and even dedicated large language models (LLMs) to improve function and variable name recovery in decompiled code. Nonetheless, these approaches alone fall short of enabling a generic LLM-based decompilation pipeline that can reliably enhance the readability and semantic clarity of decompiled outputs. In this paper, we propose BinRAG, a retrieval-augmented generation (RAG) based decompilation framework designed to enhance the decompilation of stripped binaries. Building upon name prediction models, BinRAG features three novel designs: (1) it first utilizes name prediction models to transform raw decompiled outputs into enriched, source-like queries for RAG retrieval; (2) it further enhances these queries using a fine-tuned specialized LLM conditioned on the predicted variable names, enabling more accurate retrieval from a curated example database; (3) it then integrates these retrieved examples with the target’s calling context, employing a general-purpose LLM to synthesize high-fidelity, human-readable decompiled code. Evaluation on 3,200 functions from real-world software repositories demonstrates that BinRAG improves readability by 8.4% over standard RAG and semantic precision by 31.2% over the next-best prior baseline. Our results show that BinRAG effectively scales to large codebases and significantly reduces manual effort in reverse engineering tasks.

Security patch detection (SPD) is crucial for maintaining software security, as unpatched vulnerabilities can lead to severe security risks. In recent years, learning-based SPD approaches have achieved promising results on source code. However, they are generally not directly applicable to closed-source and proprietary software, which constitute a substantial portion of real-world systems, because such software typically releases patches only in binary form while its source code remains inaccessible. Despite the impressive performance of code large language models (LLMs) on code intelligence and binary analysis tasks, such as decompilation and compiler optimization, their potential for detecting binary security patches remains largely unexplored, revealing a notable gap between their demonstrated low-level code understanding capabilities and this security-critical task. To narrow this gap, we construct a large-scale binary patch dataset comprising 19,448 samples with two representations, assembly code and pseudo-code, and systematically evaluate 19 code LLMs of varying scales for binary SPD under both prompting and fine-tuning settings. Our initial study demonstrates that directly prompting off-the-shelf code LLMs remains ineffective; even advanced prompting strategies cannot compensate for the lack of task-specific knowledge. In contrast, fine-tuning proves highly effective, with pseudo-code representation consistently yielding the best performance. The best fine-tuned model, LLM4Decompile-9B-v2, achieves 0.915 in accuracy, 0.897 in F1 score, and 0.058 in false positive rate on pseudo-code. On average, models fine-tuned on pseudo-code improve accuracy by 27.5% and F1 score by 46.1% , while reducing the false positive rate by 46.2% , relative to their assembly-based counterparts. To understand this advantage, we analyze pseudo-code and assembly code from the perspectives of embedding-space similarity and code naturalness, and find that pseudo-code more closely resembles source code, making it better aligned with the source-code-centric pretraining of code LLMs. Motivated by this finding, we further augment the pseudo-code dataset with source code data, yielding additional gains after fine-tuning, particularly for smaller models. Beyond in-distribution evaluation, we further assess representative fine-tuned code LLMs under deployment-relevant distribution shifts and find that their robustness varies across shift types. Model performance remains stable under decompiler shift and is relatively preserved under architecture shift, suggesting that fine-tuned code LLMs retain meaningful robustness under practical distribution shifts in binary security patch detection.

The WeChat mini-game ecosystem faces rampant intellectual property theft to other platforms via secondary development, yet existing JavaScript obfuscation tools are ill-equipped for large-scale applications, suffering from prohibitive processing times, severe runtime performance degradation, and unsustainable code size inflation. This paper introduces JSProtect, a high-throughput parallelized obfuscation framework designed to overcome these fundamental limitations. At the core of our framework is the Parallel-Aware Scope Analysis (PASA) algorithm, which enables two key optimizations: independent code partitioning for multi-core processing and independent namespace management that aggressively reuses short identifiers to combat code bloat. Our evaluation demonstrates that JSProtect processes 20MB codebases in minutes, maintaining 100% semantic equivalence while controlling code size inflation to as low as 20% compared to over 1,000% with baseline tools. Furthermore, it preserves near-native runtime performance and provides superior security effectiveness against both static analysis tools and large language models. This work presents a new paradigm for industrial-scale JavaScript protection that effectively balances robust security with high performance and scalability.

Code editing constitutes a fundamental practice in software development, wherein developers modify existing codebases according to natural language requirements. Accurate code editing necessitates a comprehensive understanding of both the existing codebase and the modification requirements. Although large language models (LLMs) have demonstrated promising performance in code editing tasks, they suffer from substantial inefficiency by generating entire modified files that largely consist of unchanged code. While smaller models could potentially address this inefficiency, they typically lack the capacity to effectively comprehend long code contexts required for accurate editing. To ensure both effectiveness and efficiency, we propose to decompose code editing into a two-stage cascade: edit sketch generation , wherein a large model first produces concise sketches representing the requisite modifications (the more challenging phase), and edit sketch application , wherein a smaller model integrates these sketches into the original code to produce the final output edited code (the simpler phase). This cascaded design reduces the number of tokens generated by the large model, as the majority of the output is handled by the smaller, more efficient model, thereby enhancing overall efficiency. However, the effectiveness of this approach is constrained by current small models’ limited capabilities in handling long-context scenarios and cross-file dependencies, which are essential for accurate sketch application in real-world codebases. To address these limitations and enhance smaller models’ sketch application capabilities, we introduce the first large-scale sketch application dataset comprising over 100K training instances and 800M tokens, along with a human-evaluated benchmark, and propose specialized training strategies including curriculum-based long-context training and multi-file augmentation. Our comprehensive experiments demonstrate that our cascaded framework inherently reduces inference costs compared to direct editing with large models. Furthermore, combining large models with our fine-tuned smaller models can achieve even superior performance. For instance, on the Aider benchmark, employing DeepSeek R1 as the edit sketch generation model alongside a fine-tuned Qwen2.5 Coder 14B model for the application phase improves Pass@2 11.1% compared to direct editing with DeepSeek R1 alone. Additionally, the cascaded approach reduces execution time and cost by 13% and 19%, respectively, demonstrating both performance gains and efficiency improvements.

Visual deep learning (VDL) systems power safety-critical applications like autonomous driving and demand comprehensive testing, where generating diverse image mutations is crucial. Recently, multi-modal large language models (MLLMs) introduced instruction-driven methods, offering a unified paradigm of creating diverse and complex image mutations by describing them in text. Despite the promise, their mutation quality and applicability remain largely unexplored. We present the first study assessing MLLM's adequacy in VDL testing, focusing on the semantic validity of mutated images, their alignment with text instructions (prompts), and the faithfulness in preserving unchanged semantics. Through large-scale human studies and quantitative analysis, we identify MLLMs’ potential. Notably, while SoTA MLLMs (e.g., GPT-4V) struggle with editing already-appeared semantics in images (e.g., rotating objects), they enable novel “semantic-replacement” mutations (e.g., “dress a dog with clothes” ) that were infeasible previously. These mutations bring new semantics, effectively triggering unique VDL faults. Thus, MLLM-based mutations are a vital complement to traditional methods, not a replacement. Building on our findings, we develop new testing metrics tailored to MLLM-enabled mutations and summarize lessons for employing and designing MLLM mutation pipelines in different testing scenarios. Our evaluation framework and generated dataset form a reusable benchmark for future research.

LLMs increasingly serve as general-purpose AI assistants in daily life, and their subtly unethical suggestions become a serious and real concern. It is demanding to test and mitigate such unethical suggestions from LLMs. Despite existing efforts to detect violations of “testable” facets of ethics (e.g., fairness testing), it is challenging to encode the full scope of ethics (e.g., justice, deontology) into a test oracle without human annotations or intervention. In this article, we take inspiration from reflective equilibrium, a modern moral reasoning method in moral and political philosophy, to guide our approach. Instead of seeking unethical suggestions in LLMs, we aim to identify behavioral inconsistency in LLMs’ ethics-related suggestions. These inconsistencies are anticipated to serve as a useful proxy and hint at unethical suggestions. We formulate reflective equilibrium in the form of fixed-point iteration, instantiate it as a novel test oracle, and also employ it to form a mitigation scheme for LLMs’ behavioral inconsistency on ethics-related inputs. To facilitate testing, we also create a comprehensive test suite, EthicsSuite , with 20K moral situations. In our study, we evaluate eight widely used LLMs. Our experiments reveal that LLMs are prone to ethical inconsistencies, with 81.22% of our test cases prompting ethically inconsistent suggestions on average. Our human evaluation suggests that the majority of these inconsistencies indeed manifest unethical biases. Our mitigation scheme effectively refines a significant number (80.1%) of these suggestions for commercial LLMs such as GPT-4 and Claude.