An Empirical Study on Challenges for LLM Application Developers
Top Cited Papers
Open Access
- 17 August 2025
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 34 (7) , 1-37
Abstract
In recent years, large language models (LLMs) have seen rapid advancements, significantly impacting various fields such as computer vision, natural language processing, and software engineering. These LLMs, exemplified by OpenAI's ChatGPT, have revolutionized the way we approach language understanding and generation tasks. However, in contrast to traditional software development practices, LLM development introduces new challenges for AI developers in design, implementation, and deployment. These challenges span different areas (such as prompts, APIs, and plugins), requiring developers to navigate unique methodologies and considerations specific to LLM application development. Despite the profound influence of LLMs, to the best of our knowledge, these challenges have not been thoroughly investigated in previous empirical studies. To fill this gap, we present the first comprehensive study on understanding the challenges faced by LLM developers. Specifically, we crawl and analyze 29,057 relevant questions from a popular OpenAI developer forum. We first examine their popularity and difficulty. After manually analyzing 2,364 sampled questions, we construct a taxonomy of challenges faced by LLM developers. Based on this taxonomy, we summarize a set of findings and actionable implications for LLM-related stakeholders, including developers and providers (especially the OpenAI organization).Keywords
Funding Information
- National Natural Science Foundation of China (61202006)
- Postgraduate Research & Practice Innovation Program of Jiangsu Province (SJCX24_2022)
This publication has 43 references indexed in Scilit:
- Common challenges of deep reinforcement learning applications development: an empirical studyEmpirical Software Engineering, 2024
- GRACE: Empowering LLM-based software vulnerability detection with graph structure and in-context learningJournal of Systems and Software, 2024
- Automatic smart contract comment generation via large language models and in-context learningInformation and Software Technology, 2024
- A Closer Look at Different Difficulty Levels Code Generation Abilities of ChatGPT2023
- An Empirical Study of Common Challenges in Developing Deep Learning Applications2019
- What Security Questions Do Developers Ask? A Large-Scale Study of Stack Overflow PostsJournal of Computer Science and Technology, 2016
- What are mobile developers asking about? A large scale study using stack overflowEmpirical Software Engineering, 2015
- An Empirical Study of Bugs in Machine Learning Systems2012
- A field study of API learning obstaclesEmpirical Software Engineering, 2010
- Qualitative methods in empirical studies of software engineeringIEEE Transactions on Software Engineering, 1999