Jiayang Song
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jiayang Song has a total of 29 co-authors. Here are the top co-authors by number of shared publications:
- Lei Ma
University of Tokyo, Tokyo, Japan
10 shared publications - Xuan Xie
School of Computer Science and Engineering, Macau University of Science and Technology, Macau, China
7 shared publications - Yuheng Huang
University of Tokyo, Tokyo, Japan
7 shared publications - Felix Juefei-Xu
New York University, New York, NY, USA
4 shared publications - Zhehua Zhou
University of Alberta, Edmonton, Canada
4 shared publications - Da Song
University of Alberta, Canada
3 shared publications
Latest Publications
- Open Access
- 20 August 2026
- Published by Elsevier in SSRN Electronic Journal
Test Generation based on Deep Learning: A Comprehensive Survey
Software testing is a core activity for software quality assurance. It has an indispensable role in checking the reliability, security, and performance of software systems. Among the various testing activities, Test-Case Generation (TCG) remains one of the most important yet challenging tasks. With an excellent record of success in Natural Language Processing (NLP), computer vision, and autonomous systems, Deep Learning (DL) has emerged as a transformative force in modern artificial intelligence, representing a potential paradigm shift for TCG. However, published work summarizing, classifying, and analyzing the process of Deep-Learning-based Test Generation (DLTG) remains scarce: To address this gap, this paper provides a comprehensive survey of the use of DL in TCG for software systems. We present the main phases of DLTG, identify publication trends, categorize input types, classify approaches, examine test-case types, and identify application scenarios. Currently available tools and experimental evaluations are also analyzed. We also summarize remaining key challenges and potential opportunities for future work.
- 20 August 2026
- Published by Elsevier in SSRN Electronic Journal
Open AccessTest Generation based on Deep Learning: A Comprehensive Survey
Software testing is a core activity for software quality assurance. It has an indispensable role in checking the reliability, security, and performance of software systems. Among the various testing activities, Test-Case Generation (TCG) remains one of the most important yet challenging tasks. With an excellent record of success in Natural Language Processing (NLP), computer vision, and autonomous systems, Deep Learning (DL) has emerged as a transformative force in modern artificial intelligence, representing a potential paradigm shift for TCG. However, published work summarizing, classifying, and analyzing the process of Deep-Learning-based Test Generation (DLTG) remains scarce: To address this gap, this paper provides a comprehensive survey of the use of DL in TCG for software systems. We present the main phases of DLTG, identify publication trends, categorize input types, classify approaches, examine test-case types, and identify application scenarios. Currently available tools and experimental evaluations are also analyzed. We also summarize remaining key challenges and potential opportunities for future work.
- Open Access
- 15 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Retrieval-Augmented Generation (RAG) has emerged as a critical technique for enhancing large language model (LLM) capabilities. However, practitioners face significant challenges when making RAG deployment decisions. While existing research prioritizes algorithmic innovations, a systematic gap persists in understanding fundamental engineering trade-offs that determine RAG success. We present the first comprehensive study of three universal RAG deployment decisions: whether to deploy RAG, how much information to retrieve, and how to integrate retrieved knowledge effectively. Through systematic experiments across three LLMs and six datasets spanning question answering and code generation tasks, we reveal critical insights: (1) RAG deployment must be highly selective, with variable recall thresholds and failure modes affecting up to 12.6% of samples even with perfect documents. (2) Optimal retrieval volume exhibits task-dependent behavior—QA tasks show universal patterns (5-10 documents optimal) while code generation requires scenario-specific optimization. (3) Knowledge integration effectiveness depends on task and model characteristics, with code generation benefiting significantly from prompting methods while question answering shows minimal improvement. These findings demonstrate that universal RAG strategies prove inadequate. Effective RAG systems require context-aware design decisions based on task characteristics and model capabilities. Our analysis provides evidence-based guidance for practitioners and establishes foundational insights for principled RAG deployment. Our code, data and artifacts are publicly available at https://github.com/ShengmingZ/RAG_Benchmark_Code_QA .
- 15 April 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Open AccessUnderstanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems
Retrieval-Augmented Generation (RAG) has emerged as a critical technique for enhancing large language model (LLM) capabilities. However, practitioners face significant challenges when making RAG deployment decisions. While existing research prioritizes algorithmic innovations, a systematic gap persists in understanding fundamental engineering trade-offs that determine RAG success. We present the first comprehensive study of three universal RAG deployment decisions: whether to deploy RAG, how much information to retrieve, and how to integrate retrieved knowledge effectively. Through systematic experiments across three LLMs and six datasets spanning question answering and code generation tasks, we reveal critical insights: (1) RAG deployment must be highly selective, with variable recall thresholds and failure modes affecting up to 12.6% of samples even with perfect documents. (2) Optimal retrieval volume exhibits task-dependent behavior—QA tasks show universal patterns (5-10 documents optimal) while code generation requires scenario-specific optimization. (3) Knowledge integration effectiveness depends on task and model characteristics, with code generation benefiting significantly from prompting methods while question answering shows minimal improvement. These findings demonstrate that universal RAG strategies prove inadequate. Effective RAG systems require context-aware design decisions based on task characteristics and model capabilities. Our analysis provides evidence-based guidance for practitioners and establishes foundational insights for principled RAG deployment. Our code, data and artifacts are publicly available at https://github.com/ShengmingZ/RAG_Benchmark_Code_QA .
- Open Access
- 13 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35  (3) , 1-30
AcTracer: Active Testing of Large Language Model via Multi-Stage Sampling
Performance evaluation plays a crucial role in the development lifecycle of large language models (LLMs). It estimates the model’s capability, elucidates behavior characteristics, and facilitates the identification of potential issues and limitations, thereby guiding further improvement. Given that LLMs’ diverse task-handling abilities stem from large volumes of training data, a comprehensive evaluation also necessitates abundant, well-annotated, and representative test data to assess LLM performance across various downstream tasks. However, the demand for high-quality test data often entails substantial time, computational resources, and manual efforts, sometimes causing the evaluation to be inefficient or impractical. To address these challenges, researchers propose active testing, which estimates the overall performance by selecting a subset of test data. Nevertheless, the existing active testing methods tend to be inefficient, even inapplicable, given the unique new challenges of LLMs (e.g., diverse task types, increased model complexity, and unavailability of training data). To mitigate such limitations and expedite the development cycle of LLMs, in this work, we introduce AcTracer, an active testing framework tailored for LLMs that strategically selects a small subset of test data to achieve a more accurate performance estimation for LLMs. AcTracer utilizes both internal and external information from LLMs to guide the test sampling process, reducing variance through a multi-stage pool-based active selection. Our experiment results demonstrate that AcTracer achieves state-of-the-art performance compared to existing methods across various tasks.
- 13 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35  (3) , 1-30
Open AccessAcTracer: Active Testing of Large Language Model via Multi-Stage Sampling
Performance evaluation plays a crucial role in the development lifecycle of large language models (LLMs). It estimates the model’s capability, elucidates behavior characteristics, and facilitates the identification of potential issues and limitations, thereby guiding further improvement. Given that LLMs’ diverse task-handling abilities stem from large volumes of training data, a comprehensive evaluation also necessitates abundant, well-annotated, and representative test data to assess LLM performance across various downstream tasks. However, the demand for high-quality test data often entails substantial time, computational resources, and manual efforts, sometimes causing the evaluation to be inefficient or impractical. To address these challenges, researchers propose active testing, which estimates the overall performance by selecting a subset of test data. Nevertheless, the existing active testing methods tend to be inefficient, even inapplicable, given the unique new challenges of LLMs (e.g., diverse task types, increased model complexity, and unavailability of training data). To mitigate such limitations and expedite the development cycle of LLMs, in this work, we introduce AcTracer, an active testing framework tailored for LLMs that strategically selects a small subset of test data to achieve a more accurate performance estimation for LLMs. AcTracer utilizes both internal and external information from LLMs to guide the test sampling process, reducing variance through a multi-stage pool-based active selection. Our experiment results demonstrate that AcTracer achieves state-of-the-art performance compared to existing methods across various tasks.
- Open Access
- 5 January 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35  (9) , 1-35
Antidote or Placebo? Unraveling the Efficacy of Neuron Coverage Criteria on Testing Transformer-based Language Models
In the realm of deep learning, a variety of neuron coverage criteria for Deep Neural Networks (DNNs) have been devised to effectively assess the quality of test suites and facilitate the generation of test inputs. Recently proposed coverage criteria, incorporating representation distribution and causal relationships, have infused fresh vitality into this field. However, the focus of previous works is primarily on Convolutional Neural Networks for computer vision, leading to a research gap in exploring coverage testing for language models. Concurrently, with the rise of large language models, transformer-based language models have become increasingly dominant, and numerous ones have sprouted. Therefore, the effectiveness of coverage criteria in transformer-based language tasks, especially with the introduction of novel criteria, remains an unresolved open problem. To tackle it, this study examines these concerns by evaluating a wide range of criteria, including four well-established ones and two state-of-the-art criteria, across three types of transformer-based models: encoder-only, decoder-only, and encoder-decoder models. Building on previous research, we conduct a comprehensive evaluation across three key areas: regarding test suite properties, 1) Error-revealing capability, i.e., sensitivity to adversarial examples; 2) Diversity, i.e., distribution diversity and sample fairness (category diversity); and regarding test suite generation, 3) Input generation guidance, i.e., the ability to guide the generation of more valuable samples. The experimental results demonstrate that the impact of coverage criteria is multifaceted. For the error-revealing capability of test suites, the additional coverage for erroneous samples over noise samples is only 0.32%. In terms of distribution diversity and sample fairness, 26 and 30 cases, respectively, out of 33 configurations are effectively evaluated. Additionally, incorporating neuron-wise coverage guidance during test suite generation slightly increases the production of adversarial samples by 4.56%. In conclusion, while current coverage criteria can act as an antidote for assessing simple diversity, they remain largely a placebo for the core task of revealing adversarial errors, particularly when relying on individual criterion. Consequently, their practical application requires carefully evaluating the trade-off between computational overhead and potential benefits given the massive scale of Transformers. However, this low cost-effectiveness ultimately highlights the urgent need to explore and develop more robust and efficient criteria designed specifically for Transformer-based models.
- 5 January 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35  (9) , 1-35
Open AccessAntidote or Placebo? Unraveling the Efficacy of Neuron Coverage Criteria on Testing Transformer-based Language Models
In the realm of deep learning, a variety of neuron coverage criteria for Deep Neural Networks (DNNs) have been devised to effectively assess the quality of test suites and facilitate the generation of test inputs. Recently proposed coverage criteria, incorporating representation distribution and causal relationships, have infused fresh vitality into this field. However, the focus of previous works is primarily on Convolutional Neural Networks for computer vision, leading to a research gap in exploring coverage testing for language models. Concurrently, with the rise of large language models, transformer-based language models have become increasingly dominant, and numerous ones have sprouted. Therefore, the effectiveness of coverage criteria in transformer-based language tasks, especially with the introduction of novel criteria, remains an unresolved open problem. To tackle it, this study examines these concerns by evaluating a wide range of criteria, including four well-established ones and two state-of-the-art criteria, across three types of transformer-based models: encoder-only, decoder-only, and encoder-decoder models. Building on previous research, we conduct a comprehensive evaluation across three key areas: regarding test suite properties, 1) Error-revealing capability, i.e., sensitivity to adversarial examples; 2) Diversity, i.e., distribution diversity and sample fairness (category diversity); and regarding test suite generation, 3) Input generation guidance, i.e., the ability to guide the generation of more valuable samples. The experimental results demonstrate that the impact of coverage criteria is multifaceted. For the error-revealing capability of test suites, the additional coverage for erroneous samples over noise samples is only 0.32%. In terms of distribution diversity and sample fairness, 26 and 30 cases, respectively, out of 33 configurations are effectively evaluated. Additionally, incorporating neuron-wise coverage guidance during test suite generation slightly increases the production of adversarial samples by 4.56%. In conclusion, while current coverage criteria can act as an antidote for assessing simple diversity, they remain largely a placebo for the core task of revealing adversarial errors, particularly when relying on individual criterion. Consequently, their practical application requires carefully evaluating the trade-off between computational overhead and potential benefits given the massive scale of Transformers. However, this low cost-effectiveness ultimately highlights the urgent need to explore and develop more robust and efficient criteria designed specifically for Transformer-based models.
- 30 December 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 235
LeCov: Multi-level testing criteria for large language models
- 30 December 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 235
LeCov: Multi-level testing criteria for large language models
- 29 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 7  (3) , 1626-1641
Online Safety Analysis for LLMs: A Benchmark, an Assessment, and a Path Forward
While Large Language Models (LLMs) have seen widespread applications across numerous fields, their limited interpretability poses concerns regarding their safe operations from multiple aspects, e.g., truthfulness and toxicity. Recent research has started developing quality assurance methods for LLMs, introducing techniques such as offline detectors or uncertainty estimation methods. However, these approaches mainly focus on post-generation analysis, leaving the online safety analysis for LLMs during the generation phase an unexplored area. To bridge this gap, we conduct in this work a comprehensive evaluation of the effectiveness of existing online safety analysis methods on LLMs. We begin with a pilot study that validates the feasibility of detecting unsafe outputs in the early generation process. Following this, we establish the first publicly available benchmark of online safety analysis for LLMs, including a broad spectrum of methods, models, tasks, datasets, and evaluation metrics. Utilizing this benchmark, we extensively analyze the performance of state-of-the-art online safety analysis methods on both open-source and closed-source LLMs. This analysis reveals the strengths and weaknesses of individual methods and offers valuable insights into selecting the most appropriate method based on specific application scenarios and task requirements. Furthermore, we also explore the potential of using hybridization methods, i.e., combining multiple methods to derive a collective safety conclusion, to enhance the efficacy of online safety analysis. Our findings indicate a promising direction for the development of trustworthy assurance methodologies for LLMs, facilitating their reliable deployments across diverse domains.
- 29 August 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Artificial Intelligence
- Vol. 7  (3) , 1626-1641
Online Safety Analysis for LLMs: A Benchmark, an Assessment, and a Path Forward
While Large Language Models (LLMs) have seen widespread applications across numerous fields, their limited interpretability poses concerns regarding their safe operations from multiple aspects, e.g., truthfulness and toxicity. Recent research has started developing quality assurance methods for LLMs, introducing techniques such as offline detectors or uncertainty estimation methods. However, these approaches mainly focus on post-generation analysis, leaving the online safety analysis for LLMs during the generation phase an unexplored area. To bridge this gap, we conduct in this work a comprehensive evaluation of the effectiveness of existing online safety analysis methods on LLMs. We begin with a pilot study that validates the feasibility of detecting unsafe outputs in the early generation process. Following this, we establish the first publicly available benchmark of online safety analysis for LLMs, including a broad spectrum of methods, models, tasks, datasets, and evaluation metrics. Utilizing this benchmark, we extensively analyze the performance of state-of-the-art online safety analysis methods on both open-source and closed-source LLMs. This analysis reveals the strengths and weaknesses of individual methods and offers valuable insights into selecting the most appropriate method based on specific application scenarios and task requirements. Furthermore, we also explore the potential of using hybridization methods, i.e., combining multiple methods to derive a collective safety conclusion, to enhance the efficacy of online safety analysis. Our findings indicate a promising direction for the development of trustworthy assurance methodologies for LLMs, facilitating their reliable deployments across diverse domains.
- Open Access
- 19 June 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2  (FSE) , 1615-1638
VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
The rapid advancement of generative AI and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model's performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.
- 19 June 2025
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 2  (FSE) , 1615-1638
Open AccessVLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation
The rapid advancement of generative AI and multi-modal foundation models has shown significant potential in advancing robotic manipulation. Vision-language-action (VLA) models, in particular, have emerged as a promising approach for visuomotor control by leveraging large-scale vision-language data and robot demonstrations. However, current VLA models are typically evaluated using a limited set of hand-crafted scenes, leaving their general performance and robustness in diverse scenarios largely unexplored. To address this gap, we present VLATest, a fuzzing framework designed to generate robotic manipulation scenes for testing VLA models. Based on VLATest, we conducted an empirical study to assess the performance of seven representative VLA models. Our study results revealed that current VLA models lack the robustness necessary for practical deployment. Additionally, we investigated the impact of various factors, including the number of confounding objects, lighting conditions, camera poses, unseen objects, and task instruction mutations, on the VLA model's performance. Our findings highlight the limitations of existing VLA models, emphasizing the need for further research to develop reliable and trustworthy VLA applications.
- Top Cited Papers
- 1 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51  (2) , 413-429
Look Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
The recent performance leap of Large Language Models (LLMs) opens up new opportunities across numerous industrial applications and domains. However, the potential erroneous behavior (e.g., the generation of misinformation and hallucination) has also raised severe concerns for the trustworthiness of LLMs, especially in safety-, security- and reliability-sensitive industrial scenarios, potentially hindering real-world adoptions. While uncertainty estimation has shown its potential for interpreting the prediction risks made by classic machine learning (ML) models, the unique characteristics of recent LLMs (e.g., adopting self-attention mechanism as its core, very largescale model size, often used in generative contexts) pose new challenges for the behavior analysis of LLMs. Up to the present, little progress has been made to better understand whether and to what extent uncertainty estimation can help characterize the capability boundary of an LLM, to counteract its undesired behavior, which is considered to be of great importance with the potential wide-range applications of LLMs across industry domains. To bridge the gap, in this paper, we initiate an early exploratory study of the risk assessment of LLMs from the lens of uncertainty. In particular, we conduct a large-scale study with as many as twelve uncertainty estimation methods and eight general LLMs on four NLP tasks and seven programming-capable LLMs on two code generation tasks to investigate to what extent uncertainty estimation techniques could help characterize the prediction risks of LLMs. Our findings confirm the potential of uncertainty estimation for revealing LLMs’ uncertain/nonfactual predictions. The insights derived from our study can pave the way for more advanced analysis and research on LLMs, ultimately aiming at enhancing their trustworthiness.
- 1 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 51  (2) , 413-429
Top Cited PapersLook Before You Leap: An Exploratory Study of Uncertainty Analysis for Large Language Models
The recent performance leap of Large Language Models (LLMs) opens up new opportunities across numerous industrial applications and domains. However, the potential erroneous behavior (e.g., the generation of misinformation and hallucination) has also raised severe concerns for the trustworthiness of LLMs, especially in safety-, security- and reliability-sensitive industrial scenarios, potentially hindering real-world adoptions. While uncertainty estimation has shown its potential for interpreting the prediction risks made by classic machine learning (ML) models, the unique characteristics of recent LLMs (e.g., adopting self-attention mechanism as its core, very largescale model size, often used in generative contexts) pose new challenges for the behavior analysis of LLMs. Up to the present, little progress has been made to better understand whether and to what extent uncertainty estimation can help characterize the capability boundary of an LLM, to counteract its undesired behavior, which is considered to be of great importance with the potential wide-range applications of LLMs across industry domains. To bridge the gap, in this paper, we initiate an early exploratory study of the risk assessment of LLMs from the lens of uncertainty. In particular, we conduct a large-scale study with as many as twelve uncertainty estimation methods and eight general LLMs on four NLP tasks and seven programming-capable LLMs on two code generation tasks to investigate to what extent uncertainty estimation techniques could help characterize the prediction risks of LLMs. Our findings confirm the potential of uncertainty estimation for revealing LLMs’ uncertain/nonfactual predictions. The insights derived from our study can pave the way for more advanced analysis and research on LLMs, ultimately aiming at enhancing their trustworthiness.
- 25 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36  (6) , 10678-10692
GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
Safe reinforcement learning (SRL) aims to realize a safe learning process for deep reinforcement learning (DRL) algorithms by incorporating safety constraints. However, the efficacy of SRL approaches often relies on accurate function approximations, which are notably challenging to achieve in the early learning stages due to data insufficiency. To address this issue, we introduce, in this work, a novel generalizable safety enhancer (GenSafe) that can overcome the challenge of data insufficiency and enhance the performance of SRL approaches. Leveraging model order reduction techniques, we first propose an innovative method to construct a reduced order Markov decision process (ROMDP) as a low-dimensional approximator of the original safety constraints. Then, by solving the reformulated ROMDP-based constraints, GenSafe refines the actions of the agent to increase the possibility of constraint satisfaction. Essentially, GenSafe acts as an additional safety layer for SRL algorithms. We evaluate GenSafe on multiple SRL approaches and benchmark problems. The results demonstrate its capability to improve safety performance, especially in the early learning phases, while maintaining satisfactory task performance. Our proposed GenSafe not only offers a novel measure to augment existing SRL methods but also shows broad compatibility with various SRL algorithms, making it applicable to a wide range of systems and SRL problems.
- 25 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Neural Networks and Learning Systems
- Vol. 36  (6) , 10678-10692
GenSafe: A Generalizable Safety Enhancer for Safe Reinforcement Learning Algorithms Based on Reduced Order Markov Decision Process Model
Safe reinforcement learning (SRL) aims to realize a safe learning process for deep reinforcement learning (DRL) algorithms by incorporating safety constraints. However, the efficacy of SRL approaches often relies on accurate function approximations, which are notably challenging to achieve in the early learning stages due to data insufficiency. To address this issue, we introduce, in this work, a novel generalizable safety enhancer (GenSafe) that can overcome the challenge of data insufficiency and enhance the performance of SRL approaches. Leveraging model order reduction techniques, we first propose an innovative method to construct a reduced order Markov decision process (ROMDP) as a low-dimensional approximator of the original safety constraints. Then, by solving the reformulated ROMDP-based constraints, GenSafe refines the actions of the agent to increase the possibility of constraint satisfaction. Essentially, GenSafe acts as an additional safety layer for SRL algorithms. We evaluate GenSafe on multiple SRL approaches and benchmark problems. The results demonstrate its capability to improve safety performance, especially in the early learning phases, while maintaining satisfactory task performance. Our proposed GenSafe not only offers a novel measure to augment existing SRL methods but also shows broad compatibility with various SRL algorithms, making it applicable to a wide range of systems and SRL problems.
- 18 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 50  (7) , 1921-1948
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Over the past decade, Artificial Intelligence (AI) has had great success recently and is being used in a wide range of academic and industrial fields. More recently, Large Language Models (LLMs) have made rapid advancements that have propelled AI to a new level, enabling and empowering even more diverse applications and industrial domains with intelligence, particularly in areas like software engineering and natural language processing. Nevertheless, a number of emerging trustworthiness concerns and issues exhibited in LLMs, e.g., robustness and hallucination, have already recently received much attention, without properly solving which the widespread adoption of LLMs could be greatly hindered in practice. The distinctive characteristics of LLMs, such as the self-attention mechanism, extremely large neural network scale, and autoregressive generation usage contexts, differ from classic AI software based on Convolutional Neural Networks and Recurrent Neural Networks and present new challenges for quality analysis. Up to the present, it still lacks universal and systematic analysis techniques for LLMs despite the urgent industrial demand across diverse domains. Towards bridging such a gap, in this paper, we initiate an early exploratory study and propose a universal analysis framework for LLMs, named LUNA , which is designed to be general and extensible and enables versatile analysis of LLMs from multiple quality perspectives in a human-interpretable manner. In particular, we first leverage the data from desired trustworthiness perspectives to construct an abstract model as an auxiliary analysis asset and proxy, which is empowered by various abstract model construction methods built-in LUNA . To assess the quality of the abstract model, we collect and define a number of evaluation metrics, aiming at both the abstract model level and the semantics level. Then, the semantics, which is the degree of satisfaction of the LLM w.r.t. the trustworthiness perspective, is bound to and enriches the abstract model with semantics, which enables more detailed analysis applications for diverse purposes, e.g., abnormal behavior detection. To better understand the potential usefulness of our analysis framework LUNA, we conduct a large-scale evaluation, the results of which demonstrate that 1) the abstract model has the potential to distinguish normal and abnormal behavior in LLM, 2) LUNA is effective for the real-world analysis of LLMs in practice, and the hyperparameter settings influence the performance, 3) different evaluation metrics are in different correlations with the analysis performance. In order to encourage further studies in the quality assurance of LLMs, we made all of the code and more detailed experimental results data available on the supplementary website of this paper https://sites.google.com/view/llm-luna.
- 18 June 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Software Engineering
- Vol. 50  (7) , 1921-1948
LUNA: A Model-Based Universal Analysis Framework for Large Language Models
Over the past decade, Artificial Intelligence (AI) has had great success recently and is being used in a wide range of academic and industrial fields. More recently, Large Language Models (LLMs) have made rapid advancements that have propelled AI to a new level, enabling and empowering even more diverse applications and industrial domains with intelligence, particularly in areas like software engineering and natural language processing. Nevertheless, a number of emerging trustworthiness concerns and issues exhibited in LLMs, e.g., robustness and hallucination, have already recently received much attention, without properly solving which the widespread adoption of LLMs could be greatly hindered in practice. The distinctive characteristics of LLMs, such as the self-attention mechanism, extremely large neural network scale, and autoregressive generation usage contexts, differ from classic AI software based on Convolutional Neural Networks and Recurrent Neural Networks and present new challenges for quality analysis. Up to the present, it still lacks universal and systematic analysis techniques for LLMs despite the urgent industrial demand across diverse domains. Towards bridging such a gap, in this paper, we initiate an early exploratory study and propose a universal analysis framework for LLMs, named LUNA , which is designed to be general and extensible and enables versatile analysis of LLMs from multiple quality perspectives in a human-interpretable manner. In particular, we first leverage the data from desired trustworthiness perspectives to construct an abstract model as an auxiliary analysis asset and proxy, which is empowered by various abstract model construction methods built-in LUNA . To assess the quality of the abstract model, we collect and define a number of evaluation metrics, aiming at both the abstract model level and the semantics level. Then, the semantics, which is the degree of satisfaction of the LLM w.r.t. the trustworthiness perspective, is bound to and enriches the abstract model with semantics, which enables more detailed analysis applications for diverse purposes, e.g., abnormal behavior detection. To better understand the potential usefulness of our analysis framework LUNA, we conduct a large-scale evaluation, the results of which demonstrate that 1) the abstract model has the potential to distinguish normal and abnormal behavior in LLM, 2) LUNA is effective for the real-world analysis of LLMs in practice, and the hyperparameter settings influence the performance, 3) different evaluation metrics are in different correlations with the analysis performance. In order to encourage further studies in the quality assurance of LLMs, we made all of the code and more detailed experimental results data available on the supplementary website of this paper https://sites.google.com/view/llm-luna.