Jinqiu Yang
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Jinqiu Yang has a total of 40 co-authors. Here are the top co-authors by number of shared publications:
- Tse-Hsun (Peter) Chen
Concordia University, Canada
7 shared publications - Fazle Rabbi
Concordia University, Montreal, Canada
6 shared publications - Song Wang
York University, Toronto, Ontario, Canada
5 shared publications - Junjie Li
Concordia University, Montreal, Quebec, Canada
4 shared publications - Soumit Kanti Saha
Concordia University, Montreal, Canada
3 shared publications - Dong Jae Kim
DePaul University, USA
2 shared publications
Latest Publications
- Open Access
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 2660-2683
Checked-In Secret Detection: Strings Are All You Need
Hardcoded secrets in source code pose critical security vulnerabilities which can be easily exploited by malicious adversaries. Existing regex-based detection approaches suffer from fundamental limitations, as secrets often lack identifiable patterns, resulting in poor precision and recall. Recent studies have explored context-aware detection methods, as surrounding code can reveal the purpose of candidate strings. However, these methods confront three key challenges: (1) obfuscation robustness where models over-rely on easily obfuscated identifiers, (2) cross-language generalization difficulties due to uneven training data distribution, and (3) lengthy and noisy context that introduces excessive irrelevant tokens and slows inference. We observe that strings serve as a critical information source for code semantics, offering superior contextual density, obfuscation robustness, and language independence. Based on this insight, we propose StringGroup, a novel context extraction algorithm that mines strings surrounding potential secrets. By introducing a relatively simple modification to existing patterns that narrows the analysis specifically to string literals, the method achieves significant gains. With only 33.2% of the original context, it preserves over 80% of semantic information and significantly improves the signal-to-noise ratio for secret detection. We further design a context-aware secret detection tool, Secretron, based on StringGroup methods and Transformer model. Evaluation on the SecretBench dataset demonstrates high accuracy with 98.74% F1-score and strong robustness under obfuscation and cross-language scenarios, outperforming state-of-the-art LLM-based baselines. We deploy our tool in real-world environments and successfully detect 48 previously unknown secret keys from 26 applications, demonstrating the practical effectiveness of our approach.
- 1 October 2026
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Software Engineering
- Vol. 3 (ISSTA) , 2660-2683
Open AccessChecked-In Secret Detection: Strings Are All You Need
Hardcoded secrets in source code pose critical security vulnerabilities which can be easily exploited by malicious adversaries. Existing regex-based detection approaches suffer from fundamental limitations, as secrets often lack identifiable patterns, resulting in poor precision and recall. Recent studies have explored context-aware detection methods, as surrounding code can reveal the purpose of candidate strings. However, these methods confront three key challenges: (1) obfuscation robustness where models over-rely on easily obfuscated identifiers, (2) cross-language generalization difficulties due to uneven training data distribution, and (3) lengthy and noisy context that introduces excessive irrelevant tokens and slows inference. We observe that strings serve as a critical information source for code semantics, offering superior contextual density, obfuscation robustness, and language independence. Based on this insight, we propose StringGroup, a novel context extraction algorithm that mines strings surrounding potential secrets. By introducing a relatively simple modification to existing patterns that narrows the analysis specifically to string literals, the method achieves significant gains. With only 33.2% of the original context, it preserves over 80% of semantic information and significantly improves the signal-to-noise ratio for secret detection. We further design a context-aware secret detection tool, Secretron, based on StringGroup methods and Transformer model. Evaluation on the SecretBench dataset demonstrates high accuracy with 98.74% F1-score and strong robustness under obfuscation and cross-language scenarios, outperforming state-of-the-art LLM-based baselines. We deploy our tool in real-world environments and successfully detect 48 previously unknown secret keys from 26 applications, demonstrating the practical effectiveness of our approach.
- Open Access
- 24 August 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
LLM4Log: A Systematic Review of Large Language Model-based Log Analysis
Software systems generate massive, evolving, semi-structured logs that are central to reliability engineering and AIOps, yet difficult to analyze at scale under drift and limited labels. Recent advances in pretrained Transformer models and instructiontuned large language models (LLMs) have reshaped log analysis by enabling semantic generalization and cross-source evidence integration, but also introducing deployment risks such as context limits, latency and cost, privacy constraints, and hallucinations. This paper presents LLM4Log , a systematic review of LLM-based log analysis across the end-to-end pipeline, from upstream logging-statement generation and maintenance to log parsing/structuring and downstream tasks including anomaly detection, failure prediction, root cause analysis, and log summarization. Following a structured search and manual screening protocol, we completed literature collection in November 2025 and identified 145 unique papers across seven logging tasks. We organize the research area through a unified, task-driven taxonomy, summarize common design patterns (prompting/ICL, retrieval grounding, fine-tuning, tool/agent augmentation, and verification), and analyze evaluation practices, datasets, metrics, and reproducibility. Based on these cross-paper analyses, we summarize key lessons and open challenges for reliable real-world adoption. We emphasize robustness under drift and long-tail events, grounding and faithfulness for operator-facing outputs, and deployment-oriented designs with verifiable behavior.
- 24 August 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Open AccessLLM4Log: A Systematic Review of Large Language Model-based Log Analysis
Software systems generate massive, evolving, semi-structured logs that are central to reliability engineering and AIOps, yet difficult to analyze at scale under drift and limited labels. Recent advances in pretrained Transformer models and instructiontuned large language models (LLMs) have reshaped log analysis by enabling semantic generalization and cross-source evidence integration, but also introducing deployment risks such as context limits, latency and cost, privacy constraints, and hallucinations. This paper presents LLM4Log , a systematic review of LLM-based log analysis across the end-to-end pipeline, from upstream logging-statement generation and maintenance to log parsing/structuring and downstream tasks including anomaly detection, failure prediction, root cause analysis, and log summarization. Following a structured search and manual screening protocol, we completed literature collection in November 2025 and identified 145 unique papers across seven logging tasks. We organize the research area through a unified, task-driven taxonomy, summarize common design patterns (prompting/ICL, retrieval grounding, fine-tuning, tool/agent augmentation, and verification), and analyze evaluation practices, datasets, metrics, and reproducibility. Based on these cross-paper analyses, we summarize key lessons and open challenges for reliable real-world adoption. We emphasize robustness under drift and long-tail events, grounding and faithfulness for operator-facing outputs, and deployment-oriented designs with verifiable behavior.
- Open Access
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 393-403
Engineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 393-403
Open AccessEngineering Pitfalls in AI Coding Tools: An Empirical Study of Bugs in Claude Code, Codex, and Gemini CLI
- Open Access
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 38-42
Beyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 38-42
Open AccessBeyond Translation Accuracy: Addressing False Failures in LLM-Based Code Translation
- Open Access
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 371-375
HEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 371-375
Open AccessHEJ-Robust: A Robustness Benchmark for LLM-Based Automated Program Repair
- Open Access
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 61-70
Secure-Instruct: Prompt, Synthesize, and Fine-Tune for Secure Code Generation
- 5 July 2026
- Published by Association for Computing Machinery (ACM)
- p. 61-70
Open AccessSecure-Instruct: Prompt, Synthesize, and Fine-Tune for Secure Code Generation
- Open Access
- 2 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
Understanding a program’s runtime reasoning behavior, i.e., how intermediate states and control flows lead to final execution results, is essential for reliable code generation, debugging, and automated reasoning. While large language models (LLMs) have demonstrated impressive capabilities in predicting program outputs, most prior studies have focused on output accuracy and performance comparisons, treating reasoning as a black box. As a result, models often struggle with complex control logic or subtle semantic nuances, and little is known about the structure, quality, or failure modes of their reasoning traces. Recent advances in reasoning LLMs, which explicitly generate intermediate reasoning steps before producing final answers, raise the expectation that models could not only produce correct code but also explain their reasoning in ways consistent with program semantics. This ability is particularly critical for tasks involving runtime behavior, where reasoning must capture both control flow and state changes. However, the lack of systematic evaluation of reasoning traces leaves open fundamental questions about how reasoning errors arise and how they influence execution correctness. To address this gap, we conduct the first empirical study on reasoning runtime behavior inference with reasoning LLMs to systematically uncover and characterize the errors in their reasoning traces. For our research, we curate an augmented benchmark constructed from three widely used datasets (i.e., HumanEval+, LiveCodeBench, and REPOEXEC), containing 477 code snippets. For each code snippet, we experiment with three different types of inputs, i.e., regular, edge, and invalid inputs. A total of 12 input values are selected per snippet, each paired with its ground-truth execution outcome. We evaluate four state-of-the-art reasoning LLMs, i.e., DeepSeek-R1, OpenAI o4-mini, Gemini 2.5 Flash, and Claude 4 Sonnet, and our experiment results show that the examined reasoning LLMs can achieve high accuracies ranging from 62% to 98% across all three input types. Additionally, we analyzed the errors in reasoning traces produced by LLMs and developed a comprehensive taxonomy that groups nine specific categories of inference errors under four high-level parent categories for code output inference tasks. We further investigate whether a tool-augmented reasoning approach can enhance the reasoning ability of LLMs. Using failures in the Computation Errors category as a case study, our experiments show that this method successfully corrects 69% of such errors, underscoring the promise of tool augmentation in improving LLM reasoning.
- 2 June 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
Open AccessDemystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
Understanding a program’s runtime reasoning behavior, i.e., how intermediate states and control flows lead to final execution results, is essential for reliable code generation, debugging, and automated reasoning. While large language models (LLMs) have demonstrated impressive capabilities in predicting program outputs, most prior studies have focused on output accuracy and performance comparisons, treating reasoning as a black box. As a result, models often struggle with complex control logic or subtle semantic nuances, and little is known about the structure, quality, or failure modes of their reasoning traces. Recent advances in reasoning LLMs, which explicitly generate intermediate reasoning steps before producing final answers, raise the expectation that models could not only produce correct code but also explain their reasoning in ways consistent with program semantics. This ability is particularly critical for tasks involving runtime behavior, where reasoning must capture both control flow and state changes. However, the lack of systematic evaluation of reasoning traces leaves open fundamental questions about how reasoning errors arise and how they influence execution correctness. To address this gap, we conduct the first empirical study on reasoning runtime behavior inference with reasoning LLMs to systematically uncover and characterize the errors in their reasoning traces. For our research, we curate an augmented benchmark constructed from three widely used datasets (i.e., HumanEval+, LiveCodeBench, and REPOEXEC), containing 477 code snippets. For each code snippet, we experiment with three different types of inputs, i.e., regular, edge, and invalid inputs. A total of 12 input values are selected per snippet, each paired with its ground-truth execution outcome. We evaluate four state-of-the-art reasoning LLMs, i.e., DeepSeek-R1, OpenAI o4-mini, Gemini 2.5 Flash, and Claude 4 Sonnet, and our experiment results show that the examined reasoning LLMs can achieve high accuracies ranging from 62% to 98% across all three input types. Additionally, we analyzed the errors in reasoning traces produced by LLMs and developed a comprehensive taxonomy that groups nine specific categories of inference errors under four high-level parent categories for code output inference tasks. We further investigate whether a tool-augmented reasoning approach can enhance the reasoning ability of LLMs. Using failures in the Computation Errors category as a case study, our experiments show that this method successfully corrects 69% of such errors, underscoring the promise of tool augmentation in improving LLM reasoning.
- Open Access
- 2 June 2026
- Published by Elsevier in Journal of Systems and Software
- Vol. 241
Specification-driven code translation powered by Large Language Models: How far are we?
- 2 June 2026
- Published by Elsevier in Journal of Systems and Software
- Vol. 241
Open AccessSpecification-driven code translation powered by Large Language Models: How far are we?
- 20 May 2026
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (5)
A multi-language perspective on the robustness of LLM code generation
- 20 May 2026
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (5)
A multi-language perspective on the robustness of LLM code generation
- 15 November 2025
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (1) , 16
SBEST: Spectrum-based fault localization without fault-triggering tests
- 15 November 2025
- Published by Springer Nature in Empirical Software Engineering
- Vol. 31 (1) , 16
SBEST: Spectrum-based fault localization without fault-triggering tests