Unveiling the Potential of a Conversational Agent in Developer Support: Insights from Mozilla’s PDF.js Project
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 10-18
Abstract
Large language models and other foundation models (FMs) boost productivity by automating code generation, supporting bug fixes, and generating documentation. We propose that FMs can further support Open Source Software (OSS) projects by assisting developers and guiding the community. Currently, core developers and maintainers answer queries about processes, architecture, and source code, but their time is limited, often leading to delays. To address this, we introduce DevMentorAI, a tool that enhances developer-project interactions by leveraging source code and technical documentation. DevMentorAI uses the Retrieval Augmented Generation (RAG) architecture to identify and retrieve relevant content for queries. We evaluated DevMentorAI with a case study on PDF.js project, using real questions from a development chat room and comparing the answers provided by DevMentorAI to those from humans. A Mozilla expert rated the answers, finding DevMentorAI's responses more satisfactory in 8/14 of cases, equally satisfactory in 3/14, and less satisfactory in 3/14. These results demonstrate the potential of using foundation models and the RAG approach to support developers and reduce the burden on core developers.Keywords
Funding Information
- National Science Foundation (2236198, 2247929, 2303042)
- CAPES (88887.899310\/2023-00, 88887.900069\/2023-00, 88887.915794\/2023-00)
- CAPES\/PROCAD (175956)
- CAPES\/PROEX (88887.373933\/2019-00)
- CNPq (140770\/2021-6, 141180\/2021-8, 140771\/2021-2, 315711\/2020-5, 141276\/2020-7, 141054\/2019-0)
- FAPERJ (200.510\/2023, 211.033\/2019, 202.621\/2019)
This publication has 20 references indexed in Scilit:
- Future of software development with generative AIAutomated Software Engineering, 2024
- GitHub Copilot AI pair programmer: Asset or Liability?Journal of Systems and Software, 2023
- Under the Bridge: Trolling and the Challenges of Recruiting Software Developers for Empirical Research Studies2023
- A case study of implicit mentoring, its prevalence, and impact in Apache2022
- Challenges in Survey Research2020
- We Are Family: Analyzing Communication in GitHub Software Repositories and Their Forks2020
- An Automated Big Data Accuracy Assessment Tool2019
- Social Barriers Faced by Newcomers Placing Their First Contribution in Open Source Software Projects2015
- Let's talk about it: evaluating contributions through discussion in GitHub2014
- Social coding in GitHub2012