Jiho Shin

The rapid expansion of foundation models (FMs), such as large language models (LLMs), has given rise to FMware, software systems that integrate FM(s) as core components. While building demonstration-level FMware is relatively straightforward, transitioning to production-ready systems presents numerous challenges, including reliability, high implementation costs, scalability, and compliance with privacy regulations. Our paper conducts a semi-structured thematic synthesis to identify key challenges in productionizing FMware across diverse data sources, including our industry experience developing FMArts, a FMware lifecycle engineering platform, and its integration into Huawei Cloud; grey literature; academic publications; hands-on involvement in the Open Platform for Enterprise AI (OPEA); organizing the AIware conference and bootcamp; and co-leading the ISO SPDX SBOM working group on AI and datasets. We identify critical issues in FM(s) selection, data and model alignment, prompt engineering, agent orchestration, system testing, and deployment, alongside cross-cutting concerns such as memory management, observability, and feedback integration. We discuss necessary technologies and strategies to address these challenges and offer guidance to enable the transition from demonstration systems to scalable, production-ready FMware solutions. Our findings underscore the importance of continued research and multi-industry collaboration to advance the development of production-ready FMware.

Since the release of early Large Language Models (LLMs) such as GPT-2 and GPT-3 around 2020, rapid advancements in LLM capabilities have significantly impacted the field of code intelligence, enabling automation across a wide range of coding tasks, including code generation, program repair, debugging, and software testing etc. As the LLMs are widely used in coding tasks, benchmarking their capabilities in a meticulous and meaningful way is essential. In this work, we survey 142 related papers published between January 2020 and June 2025, covering 156 unique benchmarks and 32 different coding tasks, to provide a comprehensive review of coding benchmarks to evaluate LLMs, exploring their characteristics, strengths, limitations, dataset construction, task coverage, alignment with real world challenges, and performance metrics. Our analysis indicates that Python was the most prevalent programming language, featured in 77% of the datasets, and GitHub was the most common data source, used in 46% of the cases. Also, there is an increasing trend in the publication of new benchmark datasets for coding intelligence over the past three years. Prominent publication venues are machine learning conferences such as NeurIPS, ICLR, and ICML, while comparatively fewer datasets have appeared in software engineering venues such as ICSE, ASE, and MSR. In addition, most of the benchmarks focused on the code generation task, with 86 datasets dedicated to this area. We identify key gaps and limitations in existing benchmarks, including bias in datasets, lack of standardized evaluation protocols, and insufficient coverage of real-world challenges. Finally, we present a vision for the future of LLM benchmark creation, advocating for more robust, adaptable, and better alignment with practical code intelligence demands that ensure the reliability and practicality of LLMs in coding workflows.

Modern Large Language Model (LLM)-based programming agents often rely on test execution feedback to refine their generated code. These tests are synthetically generated by LLMs. However, LLMs may produce invalid or hallucinated test cases, which can mislead feedback loops and degrade the performance of agents in refining and improving code. This paper introduces VALTEST, a novel framework that leverages semantic entropy to automatically validate test cases generated by LLMs. By analyzing the semantic structure of test cases and computing entropy-based uncertainty measures, VALTEST trains a machine learning model to classify test cases as valid or invalid and filters out invalid test cases. Experiments on multiple benchmark datasets and various LLMs show that VALTEST not only boosts test validity by up to 29% but also improves code generation performance, as evidenced by significant increases in pass@1 scores. Our extensive experiments also reveal that semantic entropy is a reliable indicator to distinguish between valid and invalid test cases and provides a robust solution for improving the correctness of LLM-generated test cases used in software testing and code generation.

Since the release of early Large Language Models (LLMs) such as GPT-2 and GPT-3 around 2020, rapid advancements in LLM capabilities have significantly impacted the field of code intelligence, enabling automation across a wide range of coding tasks, including code generation, program repair, debugging, and software testing etc. As the LLMs are widely used in coding tasks, benchmarking their capabilities in a meticulous and meaningful way is essential. In this work, we survey 142 related papers published between January 2020 and June 2025, covering 156 unique benchmarks and 32 different coding tasks, to provide a comprehensive review of coding benchmarks to evaluate LLMs, exploring their characteristics, strengths, limitations, dataset construction, task coverage, alignment with real world challenges, and performance metrics. Our analysis indicates that Python was the most prevalent programming language, featured in 77% of the datasets, and GitHub was the most common data source, used in 46% of the cases. Also, there is an increasing trend in the publication of new benchmark datasets for coding intelligence over the past three years. Prominent publication venues are machine learning conferences such as NeurIPS, ICLR, and ICML, while comparatively fewer datasets have appeared in software engineering venues such as ICSE, ASE, and MSR. In addition, most of the benchmarks focused on the code generation task, with 86 datasets dedicated to this area. We identify key gaps and limitations in existing benchmarks, including bias in datasets, lack of evolution support, lack of standardized evaluation protocols, and insufficient coverage of real-world challenges. Finally, we present a vision for the future of LLM benchmarking creation, advocating for more robust, adaptable, and better alignment with practical code intelligence demands that ensure the reliability and practicality of LLMs in coding workflows.

Static analyzers play a critical role in identifying bugs early in the software development lifecycle, but their rule implementations are often under-tested and prone to inconsistencies. To address this, we propose StaAgent, an agentic framework that harnesses the generative capabilities of Large Language Models (LLMs) to systematically evaluate static analyzer rules. StaAgent comprises four specialized agents: a Seed Generation Agent that translates bug detection rules into concrete, bug-inducing seed programs; a Code Validation Agent that ensures the correctness of these seeds; a Mutation Generation Agent that produces semantically equivalent mutants; and an Analyzer Evaluation Agent that performs metamorphic testing by comparing the static analyzer's behavior on seeds and their corresponding mutants. By revealing inconsistent behaviors, StaAgent helps uncover flaws in rule implementations. This LLM-driven, multi-agent framework offers a scalable and adaptable solution to improve the reliability of static analyzers. We evaluated StaAgent with five state-of-the-art LLMs (CodeL-lama, DeepSeek, Codestral, Qwen, and GPT-4o) across five widely used static analyzers (SpotBugs, SonarQube, ErrorProne, Infer, and PMD). The experimental results show that our approach can help reveal 64 problematic rules in the latest versions of these five static analyzers (i.e., 28 in SpotBugs, 18 in SonarQube, 6 in ErrorProne, 4 in Infer, and 8 in PMD). In addition, 53 out of the 64 bugs cannot be detected by the SOTA baseline. We have reported all the bugs to developers, with two of them already fixed. Three more have been confirmed by developers, while the rest are awaiting response. These results demonstrate the effectiveness of our approach and underscore the promise of agentic, LLM-driven data synthesis to advance software engineering.

Recently, deep learning-based test case generation approaches have been proposed to automate the generation of unit test cases. In this study, we leverage Transformer-based code models to generate unit tests with the help of Domain Adaptation (DA) at a project level. Specifically, we use CodeT5, a relatively small language model trained on source code data, and fine-tune it on the test generation task. Then, we apply domain adaptation to each target project data to learn project-specific knowledge (project-level DA). We use the Methods2test dataset to fine-tune CodeT5 for the test generation task and the Defects4j dataset for project-level domain adaptation and evaluation. We compare our approach with (a) CodeT5 fine-tuned on the test generation without DA, (b) the A3Test tool, and (c) GPT-4 on five projects from the Defects4j dataset. The results show that tests generated using DA can increase the line coverage by 18.62%, 19.88%, and 18.02% and mutation score by 16.45%, 16.01%, and 12.99% compared to the above (a), (b), and (c) baselines, respectively. The overall results show consistent improvements in metrics such as parse rate, compile rate, BLEU, and CodeBLEU. In addition, we show that our approach can be seen as a complementary solution alongside existing search-based test generation tools such as EvoSuite, to increase the overall coverage and mutation scores with an average of 34.42% and 6.8%, for line coverage and mutation score, respectively.

Recently, deep learning models have shown promising results in test oracle generation. Neural Oracle Generation (NOG) models are commonly evaluated using static (automatic) metrics which are mainly based on textual similarity of the output, e.g. BLEU, ROUGE-L, METEOR, and Accuracy. However, these textual similarity metrics may not reflect the testing effectiveness of the generated oracle within a test suite, which is often measured by dynamic (execution-based) test adequacy metrics such as code coverage and mutation score. In this work, we revisit existing oracle generation studies plus gpt-3.5 to empirically investigate the current standing of their performance in textual similarity and test adequacy metrics. Specifically, we train and run four state-of-the-art test oracle generation models on seven textual similarity and two test adequacy metrics for our analysis. We apply two different correlation analyses between these two different sets of metrics. Surprisingly, we found no significant correlation between the textual similarity metrics and test adequacy metrics. For instance, gpt-3.5 on the jackrabbit-oak project had the highest performance on all seven textual similarity metrics among the studied NOGs. However, it had the lowest test adequacy metrics compared to all the studied NOGs. We further conducted a qualitative analysis to explore the reasons behind our observations. We found that oracles with high textual similarity metrics but low test adequacy metrics tend to have complex or multiple chained method invocations within the oracle’s parameters, making them hard for the model to generate completely, affecting the test adequacy metrics. On the other hand, oracles with low textual similarity metrics but high test adequacy metrics tend to have to call different assertion types or a different method that functions similarly to the ones in the ground truth. Overall, this work complements prior studies on test oracle generation with an extensive performance evaluation on textual similarity and test adequacy metrics and provides guidelines for better assessment of deep learning applications in software test generation in the future.

Machine learning (ML) has been increasingly used in a variety of domains, while solving ML programming tasks poses unique challenges due to the fundamental difference in the nature and the construct of general programming tasks, especially for developers who do not have ML backgrounds. Automatic code generation that produces a code snippet from a natural language description can be a promising technique to accelerate ML programming tasks. In recent years, although many deep learning-based neural code generation models have been proposed with high accuracy, the fact that most of them are mainly evaluated on general programming tasks calls into question their effectiveness and usefulness in ML programming tasks. In this article, we set out to investigate the effectiveness of existing neural code generation models on ML programming tasks. For our analysis, we select six state-of-the-art neural code generation models and evaluate their performance on four widely used ML libraries, with newly created 83K pairs of natural-language described ML programming tasks. Our empirical study reveals some good, bad, and missing aspects of neural code generation models on ML tasks, with a few major ones listed below. ( Good ) Neural code generation models perform significantly better on ML tasks than on non-ML tasks with an average difference of 10.6 points in BLEU-4 scores. ( Bad ) More than 80% of the generated code is semantically incorrect. ( Bad ) Code generation models do not have significance in improving developers’ completion time. ( Good ) The generated code can help developers write correct code by providing developers with clues for using correct APIs. ( Missing ) The observation from our user study reveals the missing aspects of code generation for ML tasks, e.g., decomposing code generation for divide-and-conquer into API sequence identification and API usage generation.