Shane McIntosh

In pull-based development, Pull Requests (PRs) undergo peer review. To address review feedback, authors may "force-push" after amending, squashing, or rebasing commits. Such revisions can conflate PR-specific changes with updates on the base branch, hindering further review and introducing noise into PR-based datasets. In this paper, we empirically study 935,011 PRs from 1,445 well-maintained GitHub projects. We observe 568,218 force-push events in 242,624 PRs that span 1,366 projects. We denote the HEAD commits of the PR before and after force-pushing as a force-push pair, and observe that 210,837 pairs (37.1%) have a relationship that conflates PR changes with updates to the base branch. To systematically isolate PR-specific changes in such revisions, we propose AntiDiffamine---an ancestry-based approach that aligns commits on a common reference. AntiDiffamine can automatically deconflate 98.1% of the studied revisions, decreasing the diff size by one to three orders of magnitude. We also evaluate AntiDiffamine on fail-fix pairs, i.e., revisions that fix failing builds. A case study of Kubernetes reveals that 130 of 523 pairs have conflated diffs. Moreover, the fixing files identified when AntiDiffamine is applied overlap more with a human-labelled ground truth than when conflation is untreated or excluded. AntiDiffamine is a first attempt at addressing force-push-based diff conflation, which is having a real-world impact on practical and research workflows. To ease practical adoption, we provide a template GitHub Action, which scans PR updates for force-push events and applies AntiDiffamine to deconflate the diff.

As projects grow, the maintenance of intra- and inter-project dependencies becomes increasingly complex. If dependency maintenance is lax, redundant dependencies may accrue, inflating incremental build and test latencies. The heterogeneity of language- and tool-specific dependency expressions and the complexity of the dependency graphs that they specify exacerbate the challenge of identifying and removing redundant dependencies. To address these challenges, this paper introduces DepReduce, an automated approach for optimizing declared dependencies in artifact-based build systems. DepReduce operates directly on the dependency graph managed by the underlying build system, and formalizes the optimization objective as minimizing the cumulative rebuild cost triggered by changes to individual targets. To achieve this, DepReduce performs the dependency lifting and dependency flattening operations on the dependency graph in topological order, which we prove is both correct and optimal under the defined optimization objective. To empirically evaluate the approach, we implemented BazelDepReduce, an automated dependency optimization tool for Bazel. Bazel is an artifact-based build system with native support for multiple programming languages. We evaluated BazelDepReduce on 19 open-source Bazel projects written in seven programming languages. Among them, 16 projects across six languages achieved reductions in rebuild cost. In total, BazelDepReduce identified and removed 430 redundant dependencies, which we used to produce 16 Pull Requests (PRs). Twelve PRs have been merged by the target projects, including Angular and Apache RocketMQ, affecting up to 80.6% of subsequent commits, with a median of 26.3%. We also adapted BazelDepReduce to support Buck and Cargo, providing preliminary evidence that the implementation can be extended to other artifact-based build systems. Overall, these results show that our approach can effectively reduce rebuild cost on selected Bazel projects spanning multiple languages.

Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies.

Build systems require maintenance. Since automated quality assurance rarely targets build specifications, peer review remains the key mechanism for detecting faults; yet, previous research reports that reviews of build changes are hindered by limited tool support and the scarcity of developers with build expertise. Moreover, build changes often introduce cascading effects that propagate beyond the modified files. Such effects make it difficult for reviewers to understand the consequences of a change to build specifications, particularly without dedicated tooling. In this paper, we propose a controlled experiment to examine whether access to Build Change Impact Analysis (BCIA) enhances developer understanding and assessment of changes to build specifications. BCIA summarizes how changes propagate through build specifications via data-and control-flow relationships. We conduct this study using BuiScout, a prototype implementation of BCIA for CMake-based build systems. Using a withinsubjects experimental design, participants will perform comprehension and reasoning tasks with and without access to BuiScout. These tasks will evaluate their ability to assess change impact and reason about its implications. We will measure and compare response accuracy, explanation quality, and task completion time, along with self-reported estimates of confidence and perceived difficulty. Together, these measurements will allow us to study whether BCIA (implemented in BuiScout) enhances the effectiveness and efficiency of reviewing changes to build specifications.

Build systems require maintenance. Since automated quality assurance rarely targets build specifications, peer review remains the key mechanism for detecting faults; yet, previous research reports that reviews of build changes are hindered by limited tool support and the scarcity of developers with build expertise. Moreover, build changes often introduce cascading effects that propagate beyond the modified files. Such effects make it difficult for reviewers to understand the consequences of a change to build specifications, particularly without dedicated tooling. In this paper, we propose a controlled experiment to examine whether access to Build Change Impact Analysis (BCIA) enhances developer understanding and assessment of changes to build specifications. BCIA summarizes how changes propagate through build specifications via data-and control-flow relationships.We conduct this study using BuiScout, a prototype implementation of BCIA for CMake-based build systems. Using a withinsubjects experimental design, participants will perform comprehension and reasoning tasks with and without access to BuiScout. These tasks will evaluate their ability to assess change impact and reason about its implications. We will measure and compare response accuracy, explanation quality, and task completion time, along with self-reported estimates of confidence and perceived difficulty. Together, these measurements will allow us to study whether BCIA (implemented in BuiScout) enhances the effectiveness and efficiency of reviewing changes to build specifications.

Automated regression testing is a cornerstone of modern software development, often contributing directly to code review and Continuous Integration (CI). Yet some tests suffer from flakiness, where their outcomes vary non-deterministically. Flakiness erodes developer trust in test results, wastes computational resources, and undermines CI reliability. While prior research has examined test flakiness within individual projects, its broader ecosystem-wide impact remains largely unexplored. In this paper, we present an empirical study of test flakiness in the OpenStack ecosystem, which focuses on (1) cross-project flakiness, where flaky tests impact multiple projects, and (2) inconsistent flakiness, where a test exhibits flakiness in some projects but remains stable in others. By analyzing 649 OpenStack projects, we identify 1,535 cross-project flaky tests and 1,105 inconsistently flaky tests. We find that cross-project flakiness affects 55% of OpenStack projects and significantly increases both review time and computational costs. Surprisingly, 70% of unit tests exhibit cross-project flakiness, challenging the assumption that unit tests are inherently insulated from issues that span modules like integration and system-level tests. Through qualitative analysis, we observe that race conditions in CI, inconsistent build configurations, and dependency mismatches are the primary causes of inconsistent flakiness. These findings underline the need for better coordination across complex ecosystems, standardized CI configurations, and improved test isolation strategies.

Atoms of Confusion (AoCs) are small, syntactically valid code patterns that can increase cognitive load during program comprehension. Earlier research suggested that AoCs are common and potentially harmful, but more recent studies have questioned whether their effects generalize beyond less experienced developers. This confirmatory study aims to reexamine whether the presence of AoCs slows comprehension or alters repair preferences. Moreover, we examine whether these effects are moderated by developers’ programming experience. We investigate task completion time and the kind of repairs developers prefer when interacting with code containing AoCs. We propose a two-phase study consisting of a pre-screening questionnaire and a controlled experiment. The questionnaire will function as a qualification instrument. In the experiment, participants will complete eight Java comprehension tasks, four with an AoC and four without. For each task, developers are asked to identify a seeded defect and to rank three functionally equivalent repairs differing in AoC inclusion. Task completion time and the top-ranked repair will be analyzed using mixed-effects linear and multinomial regression models, with AoC presence as the manipulated factor and programming experience as a covariate.

Build systems require maintenance. Since automated quality assurance rarely targets build specifications, peer review remains the key mechanism for detecting faults; yet, previous research reports that reviews of build changes are hindered by limited tool support and the scarcity of developers with build expertise. Moreover, build changes often introduce cascading effects that propagate beyond the modified files. Such effects make it difficult for reviewers to understand the consequences of a change to build specifications, particularly without dedicated tooling. In this paper, we propose a controlled experiment to examine whether access to Build Change Impact Analysis (BCIA) enhances developer understanding and assessment of changes to build specifications. BCIA summarizes how changes propagate through build specifications via data-and control-flow relationships. We conduct this study using BuiScout, a prototype implementation of BCIA for CMake-based build systems. Using a within-subjects experimental design, participants will perform comprehension and reasoning tasks with and without access to BuiScout. These tasks will evaluate their ability to assess change impact and reason about its implications. We will measure and compare response accuracy, explanation quality, and task completion time, along with self-reported estimates of confidence and perceived difficulty. Together, these measurements will allow us to study whether BCIA (implemented in BuiScout) enhances the effectiveness and efficiency of reviewing changes to build specifications.

Monitoring issue tracker submissions is a crucial software maintenance activity. A key goal is the prioritization of high risk, security-related bugs. If such bugs can be recognized early, the risk of propagation to dependent products and endangerment of stakeholder benefits can be mitigated. To assist triage engineers with this task, several automatic detection techniques, from Machine Learning (ML) models to prompting Large Language Models (LLMs), have been proposed. Although promising to some extent, prior techniques often memorize lexical cues as decision shortcuts, yielding low detection rate specifically for more complex submissions. As such, these classifiers do not yet reach the practical expectations of a real-time detector of security-related issues. To address these limitations, we propose SEBERTIS, a framework to train Deep Neural Networks (DNNs) as classifiers independent of lexical cues, so that they can confidently detect fully unseen security-related issues. SEBERTIS capitalizes on fine-tuning bidirectional transformer architectures as Masked Language Models (MLMs) on a series of semantically equivalent vocabulary to prediction labels (which we call Semantic Surrogates) when they have been replaced with a mask. Our SEBERTIS-trained classifier achieves a 0.9880 F1-score in detecting security-related issues of a curated corpus of 10,000 GitHub issue reports, substantially outperforming state-of-the-art issue classifiers, with 14.44%-96.98%, 15.40%-93.07%, and 14.90%-94.72% higher detection precision, recall, and F1-score over ML-based baselines. Our classifier also substantially surpasses LLM baselines, with an improvement of 23.20%-63.71%, 36.68%-85.63%, and 39.49%-74.53% for precision, recall, and F1-score.

Design-level decisions in open-source software (OSS) projects are often made through structured mechanisms such as proposals, which require substantial community discussion and review. Despite their importance, the proposal process is resource-intensive and often leads to contributor frustration, especially when proposals are declined without clear feedback. Yet, the reasons behind proposal rejection remain poorly understood, limiting opportunities to streamline the process or guide contributors effectively. This study investigates the characteristics and outcomes of proposals in the Go programming language to understand why proposals are declined and how such outcomes might be anticipated. We conduct a mixed-method empirical study on 1,091 proposals submitted to the Go project. We quantify proposal outcomes, build a taxonomy of decline reasons, and evaluate large language models (LLMs) for predicting these outcomes. We find that proposals are more often declined than accepted, and resolution typically takes over a month. Only 14.7% of declined proposals are ever resubmitted. Through qualitative coding, we identify nine key reasons for proposal decline, such as duplication, limited use cases, or violations of project principles. This taxonomy can help contributors address issues in advance, e.g., checking for existing alternatives can reduce redundancy. We also demonstrate that GPT-based models can predict decline decisions early in the discussion (F1 score = 0.71 with partial comments), offering a practical tool for prioritizing review effort. Our findings reveal inefficiencies in the proposal process and highlight actionable opportunities for improving both contributor experience and reviewer workload by enabling early triage and guiding contributors to strengthen their proposals using a structured understanding of past decline reasons.