Engineering Practices in Open-Source Agent Frameworks: A Large-Scale Empirical Study
Open Access
- 5 July 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1573-1577
Abstract
The rapid development of Large Language Models (LLMs) accelerates the adoption of LLM-based agents in software engineering, data analysis, and automated decision-making. To support agent development, various open-source frameworks provide modular abstractions for planning, tool invocation, and memory management, lowering the engineering barrier. Although widely used, research mainly emphasizes capability modeling and architectural innovation, while systematic analysis of engineering practices and real-world usage remains limited. This paper presents a large-scale empirical study of mainstream open-source agent frameworks from an engineering perspective. We first identify the ten most popular frameworks in the open-source community and examine their popularity and language support. By analyzing official documentation, API interfaces, and source-level implementations, we extract common design principles and distill five best practices. We then formalize corresponding defect patterns and perform stratified proportional sampling on 73,014 downstream projects to evaluate how these practices are implemented. The results reveal recurring issues, including incomplete agent description, improper memory Strategy, insufficient monitoring and error handling, and suboptimal tool design. These defects appear across frameworks, indicating structural gaps between recommended abstractions and practical adoption. Overall, this study provides empirical evidence linking framework design choices to engineering risks and highlights the need for stronger guidance and support in agent framework development.Keywords
This publication has 11 references indexed in Scilit:
- Survey of Emerging Trends in LLM Agent Benchmarking2025
- Evaluation and Benchmarking of LLM Agents: A Survey2025
- Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant2025
- Using RAG, LLMs, and LangChain to Create a Query Application on Open Repositories: Demo Application on DSpace-72024
- Large Language Models for Software Engineering: A Systematic Literature ReviewACM Transactions on Software Engineering and Methodology, 2024
- ElderQA-GPT: A Large Language Model for Online Q&A on Geriatric Diseases Based on BGE Semantic Vector Knowledge Base and LangChain Architecture2024
- Chatbot Development Using LangChain: A Case Study to Foster Critical Thinking and Creativity2024
- Using an LLM to Help With Code Understanding2024
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent ConversationarXiv, 2023