What characteristics make ChatGPT effective for software issue resolution? An empirical study of task, project, and conversational signals in GitHub issues
Open Access
- 18 November 2025
- journal article
- research article
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (1) , 22
Abstract
Conversational large-language models (LLMs), such as ChatGPT, are extensively used for issue resolution tasks, particularly for generating ideas to implement new features or resolve bugs. However, not all developer-LLM conversations are useful for effective issue resolution and it is still unknown what makes some of these conversations not helpful. In this paper, we analyze 686 developer-ChatGPT conversations shared within GitHub issue threads to identify characteristics that make these conversations effective for issue resolution. First, we empirically analyze the conversations and their corresponding issue threads to distinguish helpful from unhelpful conversations. We begin by categorizing the types of tasks developers seek help with (e.g., code generation , bug identification and fixing , test generation ), to better understand the scenarios in which ChatGPT is most effective. Next, we examine a wide range of conversational, project, and issue-related metrics to uncover statistically significant factors associated with helpful conversations. Finally, we identify common deficiencies in unhelpful ChatGPT responses to highlight areas that could inform the design of more effective developer-facing tools. We found that only 62% of the ChatGPT conversations were helpful for successful issue resolution. Among different tasks related to issue resolution, ChatGPT was most helpful in assisting with code generation, and tool/library/API recommendations, but struggled with generating code explanations. Our conversational metrics reveal that helpful conversations are shorter, more readable, and exhibit higher semantic and linguistic alignment. Our project metrics reveal that larger, more popular projects and experienced developers benefit more from ChatGPT’s assistance. Our issue metrics indicate that ChatGPT is more effective on simpler issues characterized by limited developer activity and faster resolution times. These typically involve well-scoped technical problems such as compilation errors and tool feature requests. In contrast, it performs less effectively on complex issues that demand deep project-specific understanding, such as system-level code debugging and refactoring. The most common deficiencies in unhelpful ChatGPT responses include incorrect information and lack of comprehensiveness. Our findings have wide implications including guiding developers on effective interaction strategies for issue resolution, informing the development of tools or frameworks to support optimal prompt design, and providing insights on fine-tuning LLMs for issue resolution tasks.Keywords
All Related Versions
This publication has 71 references indexed in Scilit:
- GPTs are not the silver bullet: Performance and challenges of using GPTs for security bug report identificationInformation and Software Technology, 2025
- A Linguistic Comparison Between Human and ChatGPT-Generated Conversations2025
- Human-interpretable clustering of short text using large language modelsRoyal Society Open Science, 2025
- An empirical study on developers’ shared conversations with ChatGPT in GitHub pull requests and issuesEmpirical Software Engineering, 2024
- Exploring the problems, their causes and solutions of AI pair programming: A study on GitHub and Stack OverflowJournal of Systems and Software, 2024
- Are We Ready to Embrace Generative AI for Software Q&A?2023
- Finding help with programming errors: An exploratory study of novice software engineers’ focus in stack overflow postsJournal of Systems and Software, 2020
- SentiStrength-SE: Exploiting domain specificity for improved sentiment analysis in software engineering textJournal of Systems and Software, 2018
- Measuring Semantic Coherence of a Conversation2018
- Perceived language complexity in GitHub issue discussions and their effect on issue resolution2017