Dong Jae Kim
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Dong Jae Kim has a total of 14 co-authors. Here are the top co-authors by number of shared publications:
- Tse-Hsun (Peter) Chen
Concordia University, Canada
11 shared publications - Nakhla Rafi
Software PErformance, Analysis, and Reliability (SPEAR) Lab, Concordia University, Montreal, Quebec, Canada
4 shared publications - Shaowei Wang
University of Manitoba, Winnipeg, Canada
4 shared publications - Zeyang Ma
Software PErformance, Analysis, and Reliability (SPEAR) lab, Concordia University, Canada
3 shared publications - An Ran Chen
University of Alberta, Edmonton, Canada
2 shared publications - Jinqiu Yang
Concordia University, Montreal, Canada
2 shared publications
Latest Publications
- Open Access
- 20 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
LLM4FL: Multi-Agent Fault Localization via Graph Navigation and Verbal Reasoning
Locating and fixing software faults is challenging. Traditional Spectrum-Based Fault Localization (SBFL) uses test coverage and failure data but often suffers from low accuracy due to a lack of semantic understanding of code. Deep-learning approaches can improve localization, but require extensive training data and computational resources. Recent Large Language Models (LLMs) offer a promising alternative with stronger code reasoning capabilities. However, existing LLM-based fault localization techniques still struggle at the repository level due to large context windows and positional bias, in which important debugging signals may be buried in large contexts. In this work, we introduce LLM4FL , a multi-LLM-agent-based fault localization technique designed to address these challenges. LLM4FL uses three specialized agents. The Context Extraction Agent divides and prioritizes covered methods into token-fitting groups using an order-aware strategy based on Ochiai suspiciousness scores. The Debugger Agent navigates the repository through graph-based retrieval to generate failure reasoning and rank suspicious methods. The Reviewer Agent then re-evaluates and re-ranks the results through iterative self-refinement and verbal reasoning. Evaluated on Defects4J, LLM4FL improves Top-1 accuracy over prior LLM-based approaches, including AutoFL and SoapFL , by 14.5% and 4.58%, respectively. Our ablation study further highlights the importance of LLM4FL's design choices, with failure reason generation improving Top-1 performance by 12.5% and order-aware division substantially improving localization accuracy by 18.07%.
- 20 November 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Open AccessLLM4FL: Multi-Agent Fault Localization via Graph Navigation and Verbal Reasoning
Locating and fixing software faults is challenging. Traditional Spectrum-Based Fault Localization (SBFL) uses test coverage and failure data but often suffers from low accuracy due to a lack of semantic understanding of code. Deep-learning approaches can improve localization, but require extensive training data and computational resources. Recent Large Language Models (LLMs) offer a promising alternative with stronger code reasoning capabilities. However, existing LLM-based fault localization techniques still struggle at the repository level due to large context windows and positional bias, in which important debugging signals may be buried in large contexts. In this work, we introduce LLM4FL , a multi-LLM-agent-based fault localization technique designed to address these challenges. LLM4FL uses three specialized agents. The Context Extraction Agent divides and prioritizes covered methods into token-fitting groups using an order-aware strategy based on Ochiai suspiciousness scores. The Debugger Agent navigates the repository through graph-based retrieval to generate failure reasoning and rank suspicious methods. The Reviewer Agent then re-evaluates and re-ranks the results through iterative self-refinement and verbal reasoning. Evaluated on Defects4J, LLM4FL improves Top-1 accuracy over prior LLM-based approaches, including AutoFL and SoapFL , by 14.5% and 4.58%, respectively. Our ablation study further highlights the importance of LLM4FL's design choices, with failure reason generation improving Top-1 performance by 12.5% and order-aware division substantially improving localization accuracy by 18.07%.
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 144-155
Order Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 144-155
Open AccessOrder Matters! An Empirical Study on Large Language Models' Input Order Bias in Software Fault Localization
- Open Access
- 7 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (10) , 1-28
Unsupervised, Accurate, and Efficient Log Parsing Using Smaller Open-Source Large Language Models
Log parsing transforms unstructured logs into structured templates for downstream analysis. Syntax-based parsers are fast but lose accuracy on logs that deviate from predefined rules. Recently, large language models (LLMs) based log parsers have shown superior parsing accuracy but face three issues: (1) manual labeling for fine-tuning or in-context learning, (2) high cost from large volumes and limited context size of LLMs, and (3) privacy risks with commercial models. We present LibreLog, an unsupervised approach using open-source LLMs to enhance privacy and reduce cost while achieving state-of-the-art accuracy. LibreLog groups logs with a fixed-depth tree, then parses each group via: (i) similarity scoring-based retrieval augmented generation, (ii) self-reflection to refine templates, and (iii) a template memory to reduce LLM queries. On LogHub-2.0, LibreLog achieves GA 87.2, PA 85.4, FGA 82.3, and FTA 65.1, PA and FTA outperforming prior state-of-the-art LLM-based parsers by 13.7% and 6.9%, respectively. LibreLog processes all logs in 5.94 hours, a 1.7 times speedup over the fastest LLM parser. Using a larger LLM only for self-reflection further improves PA to 86.3 and FTA to 68.3 with a moderate runtime cost increase (31%). In short, LibreLog addresses privacy and cost concerns of using commercial LLMs while achieving state-of-the-art parsing efficiency and accuracy.
- 7 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (10) , 1-28
Open AccessUnsupervised, Accurate, and Efficient Log Parsing Using Smaller Open-Source Large Language Models
Log parsing transforms unstructured logs into structured templates for downstream analysis. Syntax-based parsers are fast but lose accuracy on logs that deviate from predefined rules. Recently, large language models (LLMs) based log parsers have shown superior parsing accuracy but face three issues: (1) manual labeling for fine-tuning or in-context learning, (2) high cost from large volumes and limited context size of LLMs, and (3) privacy risks with commercial models. We present LibreLog, an unsupervised approach using open-source LLMs to enhance privacy and reduce cost while achieving state-of-the-art accuracy. LibreLog groups logs with a fixed-depth tree, then parses each group via: (i) similarity scoring-based retrieval augmented generation, (ii) self-reflection to refine templates, and (iii) a template memory to reduce LLM queries. On LogHub-2.0, LibreLog achieves GA 87.2, PA 85.4, FGA 82.3, and FTA 65.1, PA and FTA outperforming prior state-of-the-art LLM-based parsers by 13.7% and 6.9%, respectively. LibreLog processes all logs in 5.94 hours, a 1.7 times speedup over the fastest LLM parser. Using a larger LLM only for self-reflection further improves PA to 86.3 and FTA to 68.3 with a moderate runtime cost increase (31%). In short, LibreLog addresses privacy and cost concerns of using commercial LLMs while achieving state-of-the-art parsing efficiency and accuracy.
- Open Access
Crash Report Enhancement with Large Language Models: An Empirical Study
Crash reports are central to software maintenance, yet many lack the diagnostic detail developers need to debug efficiently. We examine whether large language models can enhance crash reports by adding fault locations, root-cause explanations, and repair suggestions. We study two enhancement strategies: Direct-LLM, a single-shot approach that uses stack-trace context, and Agentic-LLM, an iterative approach that explores the repository for additional evidence. On a dataset of 492 real-world crash reports, LLM-enhanced reports improve Top-1 problem-localization accuracy from 10.6% (original reports) to 40.2-43.1%, and produce suggested fixes that closely resemble developer patches (CodeBLEU around 56-57%). Both our manual evaluations and LLM-as-a-judge assessment show that Agentic-LLM delivers stronger root-cause explanations and more actionable repair guidance. A user study with 16 participants further confirms that enhanced reports make crashes easier to understand and resolve, with the largest improvement in repair guidance. These results indicate that supplying LLMs with stack traces and repository code yields enhanced crash reports that are substantially more useful for debugging.
Open AccessCrash Report Enhancement with Large Language Models: An Empirical Study
Crash reports are central to software maintenance, yet many lack the diagnostic detail developers need to debug efficiently. We examine whether large language models can enhance crash reports by adding fault locations, root-cause explanations, and repair suggestions. We study two enhancement strategies: Direct-LLM, a single-shot approach that uses stack-trace context, and Agentic-LLM, an iterative approach that explores the repository for additional evidence. On a dataset of 492 real-world crash reports, LLM-enhanced reports improve Top-1 problem-localization accuracy from 10.6% (original reports) to 40.2-43.1%, and produce suggested fixes that closely resemble developer patches (CodeBLEU around 56-57%). Both our manual evaluations and LLM-as-a-judge assessment show that Agentic-LLM delivers stronger root-cause explanations and more actionable repair guidance. A user study with 16 participants further confirms that enhanced reports make crashes easier to understand and resolve, with the largest improvement in repair guidance. These results indicate that supplying LLMs with stack traces and repository code yields enhanced crash reports that are substantially more useful for debugging.
- Open Access
- 12 July 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 1 (FSE) , 1937-1959
Towards Better Graph Neural Network-Based Fault Localization through Enhanced Code Representation
Automatic software fault localization plays an important role in software quality assurance by pinpointing faulty locations for easier debugging. Coverage-based fault localization is a commonly used technique, which applies statistics on coverage spectra to rank faulty code based on suspiciousness scores. However, statisticsbased approaches based on formulae are often rigid, which calls for learning-based techniques. Amongst all, Grace , a graph-neural network (GNN) based technique has achieved state-of-the-art due to its capacity to preserve coverage spectra, i.e., test-to-source coverage relationships, as precise abstract syntax-enhanced graph representation, mitigating the limitation of other learning-based technique which compresses the feature representation. However, such representation is not scalable due to the increasing complexity of software, correlating with increasing coverage spectra and AST graph, making it challenging to extend, let alone train the graph neural network in practice. In this work, we proposed a new graph representation, DepGraph , that reduces the complexity of the graph representation by 70 % in nodes and edges by integrating the interprocedural call graph in the graph representation of the code. Moreover, we integrate additional features—code change information—into the graph as attributes so the model can leverage rich historical project data. We evaluate DepGraph using Defects4j 2.0.0, and it outperforms Grace by locating 20 % more faults in Top-1 and improving the Mean First Rank (MFR) and the Mean Average Rank (MAR) by over 50 % while decreasing GPU memory usage by 44 % and training/inference time by 85 % . Additionally, in cross-project settings, DepGraph surpasses the state-of-the-art baseline with a 42 % higher Top-1 accuracy, and 68 % and 65 % improvement in MFR and MAR, respectively. Our study demonstrates DepGraph ’s robustness, achieving state-of-the-art accuracy and scalability for future extension and adoption.
- 12 July 2024
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 1 (FSE) , 1937-1959
Open AccessTowards Better Graph Neural Network-Based Fault Localization through Enhanced Code Representation
Automatic software fault localization plays an important role in software quality assurance by pinpointing faulty locations for easier debugging. Coverage-based fault localization is a commonly used technique, which applies statistics on coverage spectra to rank faulty code based on suspiciousness scores. However, statisticsbased approaches based on formulae are often rigid, which calls for learning-based techniques. Amongst all, Grace , a graph-neural network (GNN) based technique has achieved state-of-the-art due to its capacity to preserve coverage spectra, i.e., test-to-source coverage relationships, as precise abstract syntax-enhanced graph representation, mitigating the limitation of other learning-based technique which compresses the feature representation. However, such representation is not scalable due to the increasing complexity of software, correlating with increasing coverage spectra and AST graph, making it challenging to extend, let alone train the graph neural network in practice. In this work, we proposed a new graph representation, DepGraph , that reduces the complexity of the graph representation by 70 % in nodes and edges by integrating the interprocedural call graph in the graph representation of the code. Moreover, we integrate additional features—code change information—into the graph as attributes so the model can leverage rich historical project data. We evaluate DepGraph using Defects4j 2.0.0, and it outperforms Grace by locating 20 % more faults in Top-1 and improving the Mean First Rank (MFR) and the Mean Average Rank (MAR) by over 50 % while decreasing GPU memory usage by 44 % and training/inference time by 85 % . Additionally, in cross-project settings, DepGraph surpasses the state-of-the-art baseline with a 42 % higher Top-1 accuracy, and 68 % and 65 % improvement in MFR and MAR, respectively. Our study demonstrates DepGraph ’s robustness, achieving state-of-the-art accuracy and scalability for future extension and adoption.
- Open Access
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 382-387
Decoding Anomalies! Unraveling Operational Challenges in Human-in-the-Loop Anomaly Validation
Artificial intelligence has been driving new industrial solutions for challenging problems in recent years, with many companies leveraging AI to enhance business processes and products. Automated anomaly detection emerges as one of the top priorities in AI adoption, sought after by numerous small to large-scale enterprises. Extending beyond domain-specific applications like software log analytics, where anomaly detection has perhaps garnered the most interest in software engineering, we find that very little research effort has been devoted to post-anomaly detection, such as validating anomalies. For example, validating anomalies requires human-in-the-loop interaction, though working with human experts is challenging due to uncertain requirements on how to elicit valuable feedback from them, posing formidable operationalizing challenges. In this study, we provide an experience report delving into a more holistic view of the complexities of adopting effective anomaly detection models from a requirement engineering perspective. We address challenges and provide solutions to mitigate challenges associated with operationalizing anomaly detection from diverse perspectives: inherent issues in dynamic datasets, diverse business contexts, and the dynamic interplay between human expertise and AI guidance in the decision-making process. We believe our experience report will provide insights for other companies looking to adopt anomaly detection in their own business settings.
- 10 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 382-387
Open AccessDecoding Anomalies! Unraveling Operational Challenges in Human-in-the-Loop Anomaly Validation
Artificial intelligence has been driving new industrial solutions for challenging problems in recent years, with many companies leveraging AI to enhance business processes and products. Automated anomaly detection emerges as one of the top priorities in AI adoption, sought after by numerous small to large-scale enterprises. Extending beyond domain-specific applications like software log analytics, where anomaly detection has perhaps garnered the most interest in software engineering, we find that very little research effort has been devoted to post-anomaly detection, such as validating anomalies. For example, validating anomalies requires human-in-the-loop interaction, though working with human experts is challenging due to uncertain requirements on how to elicit valuable feedback from them, posing formidable operationalizing challenges. In this study, we provide an experience report delving into a more holistic view of the complexities of adopting effective anomaly detection models from a requirement engineering perspective. We address challenges and provide solutions to mitigate challenges associated with operationalizing anomaly detection from diverse perspectives: inherent issues in dynamic datasets, diverse business contexts, and the dynamic interplay between human expertise and AI guidance in the decision-making process. We believe our experience report will provide insights for other companies looking to adopt anomaly detection in their own business settings.
- Open Access
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 382-383
Exploring the Impact of Inheritance on Test Code Maintainability
- 14 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 382-383
Open AccessExploring the Impact of Inheritance on Test Code Maintainability
- Open Access
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
A First Look at the Inheritance-Induced Redundant Test Execution
Inheritance, a fundamental aspect of object-oriented design, has been leveraged to enhance code reuse and facilitate efficient software development. However, alongside its benefits, inheritance can introduce tight coupling and complex relationships between classes, posing challenges for software maintenance. Although there are many studies on inheritance in source code, there is limited study on using inheritance in test code. In this paper, we take the first step by studying inheritance in test code, with a focus on redundant test executions caused by inherited test cases. We empirically study the prevalence of test inheritance and its characteristics. We also propose a hybrid approach that combines static and dynamic analysis to identify and locate inheritance-induced redundant test cases. Our findings reveal that (1) inheritance is widely utilized in the test code, (2) inheritance-induced redundant test executions are prevalent, accounting for 13% of all execution test cases, (3) bypassing these redundancies can help reduce 14% of the test execution time, and finally, (4) our study highlights the need for careful refactoring decisions to minimize redundant test cases and identifies the need for further research on test code quality.
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-12
Open AccessA First Look at the Inheritance-Induced Redundant Test Execution
Inheritance, a fundamental aspect of object-oriented design, has been leveraged to enhance code reuse and facilitate efficient software development. However, alongside its benefits, inheritance can introduce tight coupling and complex relationships between classes, posing challenges for software maintenance. Although there are many studies on inheritance in source code, there is limited study on using inheritance in test code. In this paper, we take the first step by studying inheritance in test code, with a focus on redundant test executions caused by inherited test cases. We empirically study the prevalence of test inheritance and its characteristics. We also propose a hybrid approach that combines static and dynamic analysis to identify and locate inheritance-induced redundant test cases. Our findings reveal that (1) inheritance is widely utilized in the test code, (2) inheritance-induced redundant test executions are prevalent, accounting for 13% of all execution test cases, (3) bypassing these redundancies can help reduce 14% of the test execution time, and finally, (4) our study highlights the need for careful refactoring decisions to minimize redundant test cases and identifies the need for further research on test code quality.
- Top Cited PapersOpen Access
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-13
LLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.
- 12 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Top Cited PapersOpen AccessLLMParser: An Exploratory Study on Using Large Language Models for Log Parsing
Logs are important in modern software development with runtime information. Log parsing is the first step in many log-based analyses, that involve extracting structured information from unstructured log data. Traditional log parsers face challenges in accurately parsing logs due to the diversity of log formats, which directly impacts the performance of downstream log-analysis tasks. In this paper, we explore the potential of using Large Language Models (LLMs) for log parsing and propose LLMParser, an LLM-based log parser based on generative LLMs and few-shot tuning. We leverage four LLMs, Flan-T5-small, Flan-T5-base, LLaMA-7B, and ChatGLM-6B in LLMParsers. Our evaluation of 16 open-source systems shows that LLMParser achieves statistically significantly higher parsing accuracy than state-of-the-art parsers (a 96% average parsing accuracy). We further conduct a comprehensive empirical analysis on the effect of training size, model size, and pre-training LLM on log parsing accuracy. We find that smaller LLMs may be more effective than more complex LLMs; for instance where Flan-T5-base achieves comparable results as LLaMA-7B with a shorter inference time. We also find that using LLMs pre-trained using logs from other systems does not always improve parsing accuracy. While using pre-trained Flan-T5-base shows an improvement in accuracy, pre-trained LLaMA results in a decrease (decrease by almost 55% in group accuracy). In short, our study provides empirical evidence for using LLMs for log parsing and highlights the limitations and future research direction of LLM-based log parsers.
- 1 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 35-46
An Empirical Study on Performance Bugs in Deep Learning Frameworks
Machine Learning (ML) and Deep Learning (DL) applications are becoming more popular due to the availability of DL frameworks such as TensorFlow and PyTorch. Therefore, the quality of DL frameworks is essential to ensure DL/ML application quality. Given the computationally expensive nature of DL tasks (e.g., training), performance is a critical aspect of DL frameworks. However, optimizing DL frameworks may have its own unique challenges due to the peculiarities of DL (e.g., hardware integration and the nature of the computation). In this paper, we conduct an empirical study on the performance bugs in DL frameworks. We conduct our study on TensorFlow and PyTorch by identifying the performance and non-performance bugs by mining the GitHub repositories. We find that 1) the proportion of newly reported performance bugs increases faster than fixed performance bugs, and the ratio of performance bugs among all bugs increases over time; 2) performance bugs take more time to fix, have larger fix sizes, and more community engagement (e.g., discussion) compared to non-performance bugs; and 3) we manually derived a taxonomy of 12 categories and 19 sub-categories of the root causes of performance bugs by studying all performance bug fixes. Finally, we present some actionable implications for researchers and developers.
- 1 October 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 35-46
An Empirical Study on Performance Bugs in Deep Learning Frameworks
Machine Learning (ML) and Deep Learning (DL) applications are becoming more popular due to the availability of DL frameworks such as TensorFlow and PyTorch. Therefore, the quality of DL frameworks is essential to ensure DL/ML application quality. Given the computationally expensive nature of DL tasks (e.g., training), performance is a critical aspect of DL frameworks. However, optimizing DL frameworks may have its own unique challenges due to the peculiarities of DL (e.g., hardware integration and the nature of the computation). In this paper, we conduct an empirical study on the performance bugs in DL frameworks. We conduct our study on TensorFlow and PyTorch by identifying the performance and non-performance bugs by mining the GitHub repositories. We find that 1) the proportion of newly reported performance bugs increases faster than fixed performance bugs, and the ratio of performance bugs among all bugs increases over time; 2) performance bugs take more time to fix, have larger fix sizes, and more community engagement (e.g., discussion) compared to non-performance bugs; and 3) we manually derived a taxonomy of 12 categories and 19 sub-categories of the root causes of performance bugs by studying all performance bug fixes. Finally, we present some actionable implications for researchers and developers.