Gengyi Sun

In pull-based development, Pull Requests (PRs) undergo peer review. To address review feedback, authors may "force-push" after amending, squashing, or rebasing commits. Such revisions can conflate PR-specific changes with updates on the base branch, hindering further review and introducing noise into PR-based datasets. In this paper, we empirically study 935,011 PRs from 1,445 well-maintained GitHub projects. We observe 568,218 force-push events in 242,624 PRs that span 1,366 projects. We denote the HEAD commits of the PR before and after force-pushing as a force-push pair, and observe that 210,837 pairs (37.1%) have a relationship that conflates PR changes with updates to the base branch. To systematically isolate PR-specific changes in such revisions, we propose AntiDiffamine---an ancestry-based approach that aligns commits on a common reference. AntiDiffamine can automatically deconflate 98.1% of the studied revisions, decreasing the diff size by one to three orders of magnitude. We also evaluate AntiDiffamine on fail-fix pairs, i.e., revisions that fix failing builds. A case study of Kubernetes reveals that 130 of 523 pairs have conflated diffs. Moreover, the fixing files identified when AntiDiffamine is applied overlap more with a human-labelled ground truth than when conflation is untreated or excluded. AntiDiffamine is a first attempt at addressing force-push-based diff conflation, which is having a real-world impact on practical and research workflows. To ease practical adoption, we provide a template GitHub Action, which scans PR updates for force-push events and applies AntiDiffamine to deconflate the diff.

Root cause analysis (RCA) is essential for diagnosing failures within complex software systems to ensure system reliability. The highly distributed and interdependent nature of modern cloud-based systems often complicates RCA efforts, particularly for multi-hop fault propagation, where symptoms appear far from their true causes. Recent advancements in Large Language Models (LLMs) present new opportunities to enhance automated RCA. However, their practical value for RCA depends on the fidelity of reasoning and decision-making. Existing work relies on historical incident corpora, operates directly on high-volume telemetry beyond current LLM capacity, or embeds reasoning inside complex multi-agent pipelines—conditions that obscure whether failures arise from reasoning itself or from peripheral design choices. We present a focused empirical evaluation that isolates an LLM’s reasoning behavior. We design a controlled experimental framework that foregrounds the LLM by using a simplified experimental setting. We evaluate six LLMs under two agentic workflows (ReAct and Plan-and-Execute) and a non-agentic baseline on two real-world case studies (GAIA and OpenRCA). In total, we executed 48,000 simulated failure scenarios, totaling 228 days of execution time. We measure both root-cause accuracy and the quality of intermediate reasoning traces. We produce a labeled taxonomy of 16 common RCA reasoning failures and use an LLM-as-a-Judge for annotation. Our results clarify where current open-source LLMs succeed and fail in multi-hop RCA, quantify sensitivity to input data modalities, and identify reasoning failures that predict final correctness. Together, these contributions provide transparent and reproducible empirical results and a failure taxonomy to guide future work on reasoning-driven system diagnosis.

Continuous Integration (CI) is a common practice adopted by modern software organizations. It plays an especially important role for large corporations like Ubisoft, where thousands of build jobs are submitted daily. Indeed, the cadence of development progress is constrained by the pace at which CI services process build jobs. To provide faster CI feedback, recent work explores how build outcomes can be anticipated. Although early results show plenty of promise, the distinct characteristics of Project X—a AAA video game project at Ubisoft—present new challenges for build outcome prediction. In the Project X setting, changes that do not modify source code also incur build failures. We also observe that the code changes that have an impact that crosses the source-data boundary are more prone to build failures than code changes that do not impact data files. Since such changes are not fully characterized by the existing set of features for build outcome prediction, state-of-the-art models tend to underperform. To incorporate the data context, we propose RavenBuild—a novel approach to build outcome prediction that leverages context-, relevance-, and dependency-aware features. In the Project X context, we observe that RavenBuild improves the F1-score of the failing class by 50%, the recall of the failing class by 105%, and the AUC by 11% with respect to the state-of-the-art BuildFast approach. To ease adoption in settings with heterogeneous project sets, we also provide a simplified alternative RavenBuild-CR, which excludes dependency-aware features. We observe across-the-board improvements when RavenBuild-CR is applied to 22 open-source projects and Project X. On the other hand, we find that a naïve Parrot approach, which simply echoes the previous build outcome as its prediction, is surprisingly competitive with BuildFast and RavenBuild. Though Parrot fails to predict when the build outcome differs from their immediate predecessor, Parrot serves well as a tendency indicator of the sequences in build outcome datasets. Thus, we recommend that future studies also compare to the Parrot approach as a baseline when evaluating build outcome prediction models.

To produce a video game, engineers and artists must iterate on the same project simultaneously. In such projects, a change to the work products of any of the teams can impact the work of other teams. As a result, any analytics tasks should consider intra- and inter-dependencies within and between artifacts produced by different teams. For instance, the focus of quality assurance teams on changes that are local to a team differs from one that impacts others. To extract and analyze such cross-disciplinary dependencies, we propose the multidisciplinary dependency graph. We instantiate our idea by developing tools that extract dependencies and construct the graph at Ubisoft---a multinational video game organization with more than 18,000 employees. Our analysis of a recently launched video game project reveals that code files only make up 2.8% of the dependency graph, and code-to-code dependencies only make up 4.3% of all dependencies. We also observe that 44% of the studied source code changes impact the artifacts that are developed by other teams, highlighting the importance of analyzing inter-artifact dependencies. A comparative analysis of cross-boundary changes with changes that do not cross boundaries indicates that cross-boundary changes are: (1) impacting a median of 120,368 files; (2) with a 51% probability of causing build failures; and (3) a 67% likelihood of introducing defects. All three measurements are larger than changes that do not cross boundaries to statistically significant degrees. We also find that cross-boundary changes are: (4) more commonly associated with gameplay functionality and feature additions that directly impact the game experience than changes that do not cross boundaries, and (5) disproportionately produced by the same team (74% of the contributors are associated with that team).