Jialun Cao

Documentation changes in mature software projects often describe newly introduced or modified behavior. This makes them a natural basis for documentation-driven feature addition, where software engineering agents implement features from public-facing documentation updates. However, existing benchmarks for agentic software development primarily rely on issue reports written for maintainer coordination and often contain implementation-level details. Motivated by this observation, this work introduces Doc2Feat-bench, a benchmark for documentation-driven feature addition. Doc2Feat-bench consists of 634 tasks across 10 mature open-source software projects, involving about 114k code changes in total. Each task pairs a documentation change with the corresponding implementation and developer-written tests, enabling evaluation of whether a software engineering agent can implement the feature described by the documentation diff. The benchmark is constructed through a five-phase pipeline, starting from release notes, and offers broad coverage across libraries, developer tools, and frameworks while preserving real-world development settings. To facilitate lightweight and reliable evaluation under limited resources, we further curate a human-validated subset named Doc2Feat-bench Verified. It contains 114 high-quality instances whose task clarity and evaluation validity are manually verified. We use Doc2Feat-bench to assess a range of state-of-the-art software engineering agents with two widely used scaffolds. Experimental results show that despite significant token consumption, the best task success rate remains as low as 37.72% (OpenHands with Qwen3-Coder-480B). Our analysis reveals that software engineering agents face key challenges in performing cross-file edits, understanding existing code modules, and correctly interpreting documentation semantics.

Automated unit test generation promises to reduce the cost of software quality assurance, and hence, is attracting attention from both academia and industry. Yet, generating assertions that are executable, meaningful to developers, and able to catch faults remains an unsolved challenge. Existing approaches either randomly enumerate assertions that are plausible based on static program analysis without considering whether they naturally fit the test prefix or query an LLM to generate assertions based on local context only, such as the test prefix and the focal method. However, we observe that local context alone is insufficient for LLMs to generate high-quality assertions because many desirable assertions are built from components that are almost impossible to guess for an LLM, such as sequences of multiple method calls. This paper presents STARS, a novel test assertion generation technique that combines the benefits of static program analysis and LLM-based synthesis. The key idea is to first gather a set of assertion components based on static program analysis and to then combine, concretize, prioritize, and improve them with an LLM. The resulting assertions go beyond what an LLM alone could realistically guess based on the test prefix and focal method, and they naturally fit the given test case. Empirical results show that STARS consistently outperforms the state-of-the-art baseline in five evaluation metrics. STARS achieves an exact-match rate of at most 83.4% using GPT-5.4. Compared with the baseline, STARS’s mutation score nearly triples that of the baseline (10.97% vs. 3.97%), approaching that of developer-written assertions, while consuming 24.75% fewer tokens and 30.6% fewer LLM queries.

In-Context Learning (ICL) has emerged as a promising solution to enhance the code generation capabilities of Large Language Models (LLMs) by incorporating code examples inside the prompt to let LLMs learn from demonstrations. However, despite their effectiveness gains, it remains unclear which specific properties of ICL-provided code examples (e.g., solution insight, essential contextual information, identifier naming styles, code formatting) drive these gains. This paper systematically investigates the impact of different sources and internal features of code examples on ICL for code generation through controlled experiments on contest-style programming questions and repository-level tasks. Our results show that while LLMs struggle to extract generalizable problem-solving insights from provided solutions to similar questions or repository snippets, their retrieval-augmented ICL performance can significantly benefit from explicit contextual information, such as input/output demonstrations, required helper functions, and namespace information. Through targeted mutation operators, we further find that identifier naming is substantially more critical than code formatting or low-level implementation details, with the elimination of descriptive variable names causing performance drops of up to 30 percentage points. Finally, we demonstrate that LLMs significantly prefer semantically meaningful identifier names and that adherence to surface-level naming conventions is far less important than semantic clarity. These findings provide practical guidelines for constructing effective ICL code examples and highlight challenges in reflection-based learning for code generation.

Automatically generating bug reproduction tests (BRT) from issue descriptions is crucial for facilitating software maintenance. Large Language Model (LLM)-based approaches have shown great potential for this task. Their effectiveness heavily relies on retrieving high-quality context from the codebase. The retrieval phase of existing approaches relies on either traditional methods like BM25 or modern LLM-driven strategies. The LLM-based retrieval strategies typically involve equipping an LLM with tools to autonomously explore the code repository or having it select the most relevant files and code snippets from a provided list as context. However, these retrieval methods suffer from three key limitations: (1) They often employ a unified strategy for retrieving both source code and test cases, overlooking their distinct retrieval requirements. (2) They focus solely on semantic similarity, ignoring function call relationships that reflect behavioral relevance, which often leads to the retrieval of irrelevant context. (3) The retrieval lacks a feedback loop from the generation phase, preventing it from refining the context based on execution results. These limitations collectively result in low-quality context, thereby hindering the accuracy of bug reproduction. To address these challenges, we propose iCoRe, an iterative, correlation-aware context retrieval approach. iCoRe is explicitly designed to be aware of three key correlations: 1) the correlation between source code and test cases, which requires differentiated retrieval, 2) the correlation between textual semantics and function call structures for accurate relevance assessment, and 3) the correlation between the retrieval and generation phases, which enables iterative feedback and refinement. To evaluate iCoRe, we integrate it with an LLM-based BRT generator and conduct a comprehensive evaluation on the SWT-bench Lite and TDD-bench Verified benchmarks. Experimental results show that our method achieves a Fail-to-Pass rate of 42.0% and 52.8% respectively, representing significant 19.7%--31.7% relative improvements over existing retrieval methods.

Model checking in TLA+ provides strong correctness guarantees, yet practitioners continue to face significant challenges in interpreting counterexamples, understanding large state-transition graphs, and repairing faulty models. These difficulties stem from the limited explainability of raw model-checker output and the substantial manual effort required to trace violations back to source specifications. Although the TLA+ Toolbox includes a state diagram viewer, it offers only a static, fully expanded graph without folding, color highlighting, or semantic explanations, which limits its scalability and interpretability. We present ModelWisdom , an interactive environment that uses visualization and large language models to make TLA+ model checking more interpretable and actionable. ModelWisdom offers: (i) Model Visualization, with colorized violation highlighting, click-through links from transitions to TLA+ code, and mapping between violating states and broken properties; (ii) Graph Optimization, including tree-based structuring and node/edge folding to manage large models; (iii) Model Digest, which summarizes and explains subgraphs via large language models (LLMs) and performs preprocessing and partial explanations; and (iv) Model Repair, which extracts error information and supports iterative debugging. Together, these capabilities turn raw model-checker output into an interactive, explainable workflow, improving understanding and reducing debugging effort for nontrivial TLA+ specifications. This tool is available: https://github.com/ModelWisdom/ModelWisdom . A demonstrative video can be found at https://www.youtube.com/watch?v=plyZo30VShA .