Unveiling ChatGPT's Usage in Open Source Projects: A Mining-based Study
Open Access
- 15 April 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 571-583
Abstract
Large Language Models (LLMs) have gained significant attention in the software engineering community. Nowadays developers have the possibility to exploit these models through industrial-grade tools providing a handy interface toward LLMs, such as OpenAI's ChatGPT. While the potential of LLMs in assisting developers across several tasks has been documented in the literature, there is a lack of empirical evidence mapping the actual usage of LLMs in software projects. In this work, we aim at filling such a gap. First, we mine 1,501 commits, pull requests (PRs), and issues from open-source projects by matching regular expressions likely to indicate the usage of ChatGPT to accomplish the task. Then, we manually analyze these instances, discarding false positives (i.e., instances in which ChatGPT was mentioned but not actually used) and categorizing the task automated in the 467 true positive instances (165 commits, 159 PRs, 143 issues). This resulted in a taxonomy of 45 tasks which developers automate via ChatGPT. The taxonomy, accompanied with representative examples, provides (i) developers with valuable insights on how to exploit LLMs in their workflow and (ii) researchers with a clear overview of tasks that, according to developers, could benefit from automated solutions.Keywords
All Related Versions
Funding Information
- European Research Council (851720)
- PRIN 2020 (2020W3A5FY)
This publication has 60 references indexed in Scilit:
- Translating Video Recordings of Complex Mobile App UI Gestures into Replayable ScenariosIEEE Transactions on Software Engineering, 2022
- Enhancing Mobile App Bug Reporting via Real-Time Understanding of Reproduction StepsIEEE Transactions on Software Engineering, 2022
- PRHAN: Automated Pull Request Description Generation Based on Hybrid Attention NetworkJournal of Systems and Software, 2022
- Neural Transfer Learning for Repairing Security Vulnerabilities in C CodeIEEE Transactions on Software Engineering, 2022
- Studying test-driven development and its retainment over a six-month time spanJournal of Systems and Software, 2021
- Automated Documentation of Android AppsIEEE Transactions on Software Engineering, 2019
- Listening to the Crowd for the Release Planning of Mobile AppsIEEE Transactions on Software Engineering, 2017
- A Dissection of the Test-Driven Development Process: Does It Really Matter to Test-First or to Test-Last?IEEE Transactions on Software Engineering, 2016
- Improving Automated Bug Triaging with Specialized Topic ModelIEEE Transactions on Software Engineering, 2016
- Recommendation Systems for Software EngineeringIEEE Software, 2010