Ali Asgari
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Ali Asgari has a total of 15 co-authors. Here are the top co-authors by number of shared publications:
- Annibale Panichella
Delft University of Technology, Delft, Netherlands
4 shared publications - Mitchell Olsthoorn
Delft University of Technology, Delft, Netherlands
2 shared publications - Pouria Derakhshanfar
JetBrains Research, Amsterdam, Netherlands
2 shared publications Annibale Panichella
1 shared publication- Antonio Guerriero
University of Salerno
1 shared publication - Ashkan Sami
Edinburgh Napier University
1 shared publication
Latest Publications
- Open Access
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 3316-3339
Test Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code (Replicability Study)
Test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs) under limited testing budgets, where labeling cost is a primary concern and early model failure detection is a key objective. Although prior studies report promising results, existing empirical evaluations focus almost exclusively on vision-based DNNs and datasets. As observed in recent surveys, models and datasets specifically designed for software engineering tasks have not been considered, leaving it unclear whether prior findings generalize to LLM code models. This paper presents a large-scale replication study of TCS techniques in the context of LLM code models. We re-examine established TCS strategies originally proposed for DNNs and complement them with statistical sampling strategies that have not previously been evaluated for TCS. We assess their effectiveness on three code-related classification tasks: clone detection, vulnerability detection, and technical debt prediction. The study spans 17 task-specific fine-tuned model instances, 7 predictive features, and 13 selection strategies, including 12 feature-aware strategies and simple random sampling (SRS) as a feature-agnostic baseline. We evaluate performance along two dimensions: operational accuracy estimation and early failure discovery. The results indicate that only a subset of findings reported for vision-based DNNs generalize when TCS is applied to LLMs for code. In particular, uncertainty-based features are effective for early failure discovery, while representation-based features are more robust for accuracy estimation. At the same time, performance varies substantially across tasks and models, indicating that the effectiveness of TCS techniques is context-dependent. Overall, this study provides empirical evidence on the replicability of TCS techniques beyond vision-based deep learning and offers insights into their use for the operational evaluation of LLMs for code.
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 3316-3339
Open AccessTest Case Selection for Deep Neural Networks: A Replication Study on LLMs for Code (Replicability Study)
Test case selection (TCS) techniques have been explored to support the operational evaluation of deep neural networks (DNNs) under limited testing budgets, where labeling cost is a primary concern and early model failure detection is a key objective. Although prior studies report promising results, existing empirical evaluations focus almost exclusively on vision-based DNNs and datasets. As observed in recent surveys, models and datasets specifically designed for software engineering tasks have not been considered, leaving it unclear whether prior findings generalize to LLM code models. This paper presents a large-scale replication study of TCS techniques in the context of LLM code models. We re-examine established TCS strategies originally proposed for DNNs and complement them with statistical sampling strategies that have not previously been evaluated for TCS. We assess their effectiveness on three code-related classification tasks: clone detection, vulnerability detection, and technical debt prediction. The study spans 17 task-specific fine-tuned model instances, 7 predictive features, and 13 selection strategies, including 12 feature-aware strategies and simple random sampling (SRS) as a feature-agnostic baseline. We evaluate performance along two dimensions: operational accuracy estimation and early failure discovery. The results indicate that only a subset of findings reported for vision-based DNNs generalize when TCS is applied to LLMs for code. In particular, uncertainty-based features are effective for early failure discovery, while representation-based features are more robust for accuracy estimation. At the same time, performance varies substantially across tasks and models, indicating that the effectiveness of TCS techniques is context-dependent. Overall, this study provides empirical evidence on the replicability of TCS techniques beyond vision-based deep learning and offers insights into their use for the operational evaluation of LLMs for code.
- Open Access
- 12 September 2026
- Published by Elsevier in Journal of Systems and Software
- Vol. 243
What challenges do developers face in AI agent systems? An empirical study on Stack Overflow & GitHub issues
- 12 September 2026
- Published by Elsevier in Journal of Systems and Software
- Vol. 243
Open AccessWhat challenges do developers face in AI agent systems? An empirical study on Stack Overflow & GitHub issues
- Open Access
- 13 February 2026
- Published by Elsevier in SSRN Electronic Journal
What Challenges Do Developers Face in AI Agent Systems? An Empirical Study on Stack Overflow & GitHub Issues
AI Agents have rapidly gained prominence in both research and industry as systems that extend large language models with planning, tool use, memory, and goal-directed action. Despite this progress, the development and maintenance of Agent systems present recurring engineering difficulties that are not yet well characterized in developer-facing evidence. To address this gap, this study analyzes developer discussions on Stack Overflow and failure reports from GitHub issue trackers associated with widely used Agent frameworks. For Stack Overflow, an Agent-focused corpus is constructed through tag expansion and filtering, latent themes are derived using LDA-MALLET, and topics are manually validated and labeled. For GitHub, a taxonomy of issue themes is developed to capture deployment-time failures and maintenance burdens. Analysis across both platforms identifies seven Stack Overflow topics (comprising 28 subtopics) and thirteen GitHub issue topics, which are synthesized into five overarching families of major challenges related to developing and maintaining AI agents: (1) environment, platforms, and dependency management; (2) retrieval, embeddings, and Agent memory; (3) orchestration and execution control; (4) interaction contracts between models and tools; and (5) runtime reliability and operational robustness. Topic popularity and difficulty are quantified, revealing that widely discussed issues, such as installation and prompting, are often resolved more quickly, whereas retrieval- and orchestration-related challenges are less visible, more complex, and tend to persist as ongoing maintenance burdens on GitHub.
- 13 February 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessWhat Challenges Do Developers Face in AI Agent Systems? An Empirical Study on Stack Overflow & GitHub Issues
AI Agents have rapidly gained prominence in both research and industry as systems that extend large language models with planning, tool use, memory, and goal-directed action. Despite this progress, the development and maintenance of Agent systems present recurring engineering difficulties that are not yet well characterized in developer-facing evidence. To address this gap, this study analyzes developer discussions on Stack Overflow and failure reports from GitHub issue trackers associated with widely used Agent frameworks. For Stack Overflow, an Agent-focused corpus is constructed through tag expansion and filtering, latent themes are derived using LDA-MALLET, and topics are manually validated and labeled. For GitHub, a taxonomy of issue themes is developed to capture deployment-time failures and maintenance burdens. Analysis across both platforms identifies seven Stack Overflow topics (comprising 28 subtopics) and thirteen GitHub issue topics, which are synthesized into five overarching families of major challenges related to developing and maintaining AI agents: (1) environment, platforms, and dependency management; (2) retrieval, embeddings, and Agent memory; (3) orchestration and execution control; (4) interaction contracts between models and tools; and (5) runtime reliability and operational robustness. Topic popularity and difficulty are quantified, revealing that widely discussed issues, such as installation and prompting, are often resolved more quickly, whereas retrieval- and orchestration-related challenges are less visible, more complex, and tend to persist as ongoing maintenance burdens on GitHub.
- Open Access
- 20 October 2025
- Published by Springer Nature in Empirical Software Engineering
- Vol. 30 (6) , 1-57
Exploring the black box: analysing explainable AI challenges and best practices through stack exchange discussions
Explainable Artificial Intelligence (XAI) is a crucial domain within research and industry, aiming to develop AI models that provide human-understandable explanations for their decisions. While the challenges in AI, deep learning, and big data have been extensively explored, the specific concerns of XAI developers have received limited attention. To address this gap, we analysed discussions on Stack Exchange websites to delve into these issues. Through a combination of automated and Manual analysis, we identified 6 overarching categories, 10 distinct topics, and 40 sub-topics commonly discussed by developers. Our examination revealed a steady rise in discussions on XAI since late 2015, initially focusing on conceptualisation and practical applications, with a notable surge in activity across all topic categories since 2019. Notably, Concepts and Applications, Tools Troubleshooting, and Neural Networks Interpretation emerged as the most popular topics. Troubleshooting challenges were commonly encountered with tools like SHAP, ELI5, and AIF360, while visualisation issues were prevalent with Yellowbrick and SHAP. Furthermore, our analysis suggests that addressing questions related to XAI poses greater difficulty compared to other machine-learning questions.
- 20 October 2025
- Published by Springer Nature in Empirical Software Engineering
- Vol. 30 (6) , 1-57
Open AccessExploring the black box: analysing explainable AI challenges and best practices through stack exchange discussions
Explainable Artificial Intelligence (XAI) is a crucial domain within research and industry, aiming to develop AI models that provide human-understandable explanations for their decisions. While the challenges in AI, deep learning, and big data have been extensively explored, the specific concerns of XAI developers have received limited attention. To address this gap, we analysed discussions on Stack Exchange websites to delve into these issues. Through a combination of automated and Manual analysis, we identified 6 overarching categories, 10 distinct topics, and 40 sub-topics commonly discussed by developers. Our examination revealed a steady rise in discussions on XAI since late 2015, initially focusing on conceptualisation and practical applications, with a notable surge in activity across all topic categories since 2019. Notably, Concepts and Applications, Tools Troubleshooting, and Neural Networks Interpretation emerged as the most popular topics. Troubleshooting challenges were commonly encountered with tools like SHAP, ELI5, and AIF360, while visualisation issues were prevalent with Yellowbrick and SHAP. Furthermore, our analysis suggests that addressing questions related to XAI poses greater difficulty compared to other machine-learning questions.
- Open Access
- 9 September 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (9) , 1-54
Metamorphic Testing of Deep Code Models: A Systematic Literature Review
Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models’ robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.
- 9 September 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (9) , 1-54
Open AccessMetamorphic Testing of Deep Code Models: A Systematic Literature Review
Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models’ robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.
- Open Access
Metamorphic Testing of Deep Code Models: A Systematic Literature Review
Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models' robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.
Open AccessMetamorphic Testing of Deep Code Models: A Systematic Literature Review
Large language models and deep learning models designed for code intelligence have revolutionized the software engineering field due to their ability to perform various code-related tasks. These models can process source code and software artifacts with high accuracy in tasks such as code completion, defect detection, and code summarization; therefore, they can potentially become an integral part of modern software engineering practices. Despite these capabilities, robustness remains a critical quality attribute for deep-code models as they may produce different results under varied and adversarial conditions (e.g., variable renaming). Metamorphic testing has become a widely used approach to evaluate models' robustness by applying semantic-preserving transformations to input programs and analyzing the stability of model outputs. While prior research has explored testing deep learning models, this systematic literature review focuses specifically on metamorphic testing for deep code models. By studying 45 primary papers, we analyze the transformations, techniques, and evaluation methods used to assess robustness. Our review summarizes the current landscape, identifying frequently evaluated models, programming tasks, datasets, target languages, and evaluation metrics, and highlights key challenges and future directions for advancing the field.
- Open Access
- 25 December 2024
- Published by Springer Nature in Research Square
From Testing to Evaluation of NLP and LLM Systems: An Analysis of Researchers and Practitioners Perspectives through Systematic Literature Review and Developers’ Community Platforms Mining
Natural Language Processing (NLP) systems and Large Language Models (LLMs) are core components of applications driven by Artificial Intelligence, today widely spread in many fields including healthcare, finance, legal services. The evaluation of quality attributes of NLP/LLM-based systems is currently of great interest of researchers as well as of professionals. This study presents an analysis of the evaluation of quality attributes of NLP/LLM-based systems in the perspectives of both researchers and practitioners. The former is based on a systematic review of the scientific literature. The latter is based on data mined from professional platforms (StackOverflow, DataScience Stack-Exchange, AI StackExchange) used by developers’ communities to share knowledge and technical solutions. The systematic literature review features a quantitative analysis of: 1) the quality attributes that researchers target in their NLP/LLMs evaluation studies; 2) the tasks, datasets and models used, and 3) the evaluation methods employed in research studies. The mining of discussion data from professional platforms features a trend analysis of the significance of quality attributes and of the difficulty of their evaluation in the developers’ perspective. Overall, the comparison between researchers’ and practitioners’ perspectives provides insights on how challenges related to the evaluation of NLP/LLM quality attributes are addressed, reveals several interesting differences, and draws useful implications for both.
- 25 December 2024
- Published by Springer Nature in Research Square
Open AccessFrom Testing to Evaluation of NLP and LLM Systems: An Analysis of Researchers and Practitioners Perspectives through Systematic Literature Review and Developers’ Community Platforms Mining
Natural Language Processing (NLP) systems and Large Language Models (LLMs) are core components of applications driven by Artificial Intelligence, today widely spread in many fields including healthcare, finance, legal services. The evaluation of quality attributes of NLP/LLM-based systems is currently of great interest of researchers as well as of professionals. This study presents an analysis of the evaluation of quality attributes of NLP/LLM-based systems in the perspectives of both researchers and practitioners. The former is based on a systematic review of the scientific literature. The latter is based on data mined from professional platforms (StackOverflow, DataScience Stack-Exchange, AI StackExchange) used by developers’ communities to share knowledge and technical solutions. The systematic literature review features a quantitative analysis of: 1) the quality attributes that researchers target in their NLP/LLMs evaluation studies; 2) the tasks, datasets and models used, and 3) the evaluation methods employed in research studies. The mining of discussion data from professional platforms features a trend analysis of the significance of quality attributes and of the difficulty of their evaluation in the developers’ perspective. Overall, the comparison between researchers’ and practitioners’ perspectives provides insights on how challenges related to the evaluation of NLP/LLM quality attributes are addressed, reveals several interesting differences, and draws useful implications for both.