Automated Insights Into GitHub Collaboration Dynamics
Open Access
- 2 May 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 85526-85542
Abstract
Today, GitHub™ is the most widely used platform for open-source software development. Large projects may comprise hundreds of distributed collaborators and thousands of GitHub “issues” (structured discussions). It includes basic support for dealing with issues, via “Pull Requests” (PRs)—document changes that can manually be defined to “close” them (i.e., they address and thereby conclude the issue discussion). Unresolved issues can pile up. For example, at the time of writing, the Kubernetes repository has almost 2000 open issues; finding which ones a PR might close is a hard task by itself. We address this by automatically identifying issue-PR relationships using language models (LMs) and leveraging Information Retrieval (IR) techniques. To foster further research, we contribute a carefully curated novel dataset called CodeConvo, reflecting the most influential open-source repositories for code development as well as some technical document repositories. We use this dataset to benchmark several state-of-theart (SOTA) non-proprietary models that show exceptional performance on the MTEB benchmark, as well as to train and evaluate the performance of Smart Insights into GitHub Issue-PR Relations (SIGIR), our tailored model. The best SIGIR model/data combination yields an average Mean Reciprocal Rank (MRR) above 0.7, around 20% higher than the best baseline performance. Notably, ablation studies revealed that knowledge transfer occurs not only between different programming languages but also between code and technical documents, albeit to a lesser extent. a We believe that these results are encouraging and that they can stimulate the practical application of LMs for taming the complexity of very large projects, in GitHub and beyond. The models and dataset of this study can be found at Huggingface.Keywords
This publication has 12 references indexed in Scilit:
- Natural Language Reasoning, A SurveyACM Computing Surveys, 2024
- Revealing Software Development Work Patterns with PR-Issue Graph TopologiesProceedings of the ACM on Software Engineering, 2024
- Mitigating the Impact of False Negative in Dense Retrieval with Contrastive Confidence RegularizationProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Generating representative, live network traffic out of millions of code repositories2022
- Automated unearthing of dangerous issue reports2022
- How different are different diff algorithms in Git?Empirical Software Engineering, 2019
- How Are Issue Units Linked? Empirical Study on the Linking Behavior in GitHub2018
- Version Control System: A ReviewProcedia Computer Science, 2018
- The Probabilistic Relevance Framework: BM25 and BeyondFoundations and Trends® in Information Retrieval, 2009
- The Internet Standards Process -- Revision 31996