What Happened in This Pipeline? Diffing Build Logs with CiDiff
Open Access
- 22 June 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2 (ISSTA) , 2023-2044
Abstract
Continuous integration (CI) is widely used by developers to ensure the quality and reliability of their software projects. However, diagnosing a CI regression is a tedious process that involves the manual analysis of lengthy build logs. In this paper, we explore how textual differencing can support the debugging of CI regressions. As off-the-shelf diff algorithms produce suboptimal results, in this work we introduce a new diff algorithm specifically tailored to build logs called CiDiff. We evaluate CiDiff against several baselines on a novel dataset of 17 906 CI regressions, performing an accuracy study, a quantitative study and a user-study. Notably, our algorithm reduces the number of lines to inspect by about 60 % in the median case, with reasonable overhead compared to the state-of-practice LCS-diff. Finally, our algorithm is preferred by the majority of participants in 70 % of the regression cases, whereas LCS-diff is preferred in only 5 % of the casesKeywords
All Related Versions
Funding Information
- Agence Nationale de la Recherche (ANR?21?CE25?0007)
This publication has 35 references indexed in Scilit:
- Landscape of Automated Log Analysis: A Systematic Literature Review and Mapping StudyIEEE Access, 2022
- Statistical Log Differencing2019
- srcDiff: A syntactic differencing approach to improve the understandability of deltasJournal of Software: Evolution and Process, 2019
- Every build you break: developer-oriented assistance for build failure resolutionEmpirical Software Engineering, 2019
- Tools and Benchmarks for Automated Log Parsing2019
- Ldiff: An enhanced line differencing tool2009
- Edit distance with move operationsJournal of Discrete Algorithms, 2007
- Mauve: Multiple Alignment of Conserved Genomic Sequence With RearrangementsGenome Research, 2004
- Sample Size for Estimating Multinomial ProportionsThe American Statistician, 1987
- AnO(ND) difference algorithm and its variationsAlgorithmica, 1986