Ruiyin Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Ruiyin Li has a total of 30 co-authors. Here are the top co-authors by number of shared publications:
- Peng Liang
School of Computer Science, Wuhan University, Wuhan, China
6 shared publications - Paris Avgeriou
University of Groningen
3 shared publications - Weisong Sun
Nanyang technological university, Singapore, Singapore
2 shared publications - Zengyang Li
School of Computer Science, Central China Normal University, WuHan, HuBei, China
2 shared publications - Beiqi Zhang
School of Computer Science, Wuhan University, China
1 shared publication - Chen Yang
School of Artificial Intelligence, Shenzhen Polytechnic University, China
1 shared publication
Latest Publications
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2096-2108
Out of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2096-2108
Open AccessOut of Distribution, Out of Luck: How Well Can LLMs Trained on Vulnerability Datasets Detect Top 25 CWE Weaknesses?
- Top Cited PapersOpen Access
- 20 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (10) , 1-30
Unveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub
The advent of Large Language Models (LLMs) has introduced a new paradigm in Software Engineering (SE), with generative AI tools like ChatGPT gaining widespread adoption among developers. While ChatGPT’s potential has been extensively discussed, empirical evidence about how developers actually use LLMs’ assistance in real-world practices remains limited. To bridge this gap, we conducted a large-scale empirical analysis of ChatGPT usage on GitHub, and we presented DevChat, a curated dataset of 2,547 publicly shared ChatGPT conversation links collected from GitHub between May 2023 and June 2024. Through comprehensively analyzing DevChat, we explored the characteristics of developer–ChatGPT interaction patterns and identified five key categories of developers’ purposes of using developer-ChatGPT conversations during software development. Additionally, we investigated the dominant development-related activities in which ChatGPT is used, and presented a mapping framework that links GitHub data sources, development-related activities, and SE tasks. The findings show that interactions are typically short and task-focused (most are 1-3 turns); developers share conversations mainly to delegate tasks, resolve problems, and acquire knowledge, revealing five purpose categories; ChatGPT is most frequently engaged for Software Implementation and Maintenance & Evolution ; we identified 39 fine-grained SE tasks supported by ChatGPT, with Code Generation & Completion as well as Code modification & Optimization being the most prominent. Our study offers a comprehensive mapping of ChatGPT’s applications in real-world software development scenarios and provides a foundation for understanding LLMs’ practical roles in software development.
- 20 February 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (10) , 1-30
Top Cited PapersOpen AccessUnveiling the Role of ChatGPT in Software Development: Insights from Developer-ChatGPT Interactions on GitHub
The advent of Large Language Models (LLMs) has introduced a new paradigm in Software Engineering (SE), with generative AI tools like ChatGPT gaining widespread adoption among developers. While ChatGPT’s potential has been extensively discussed, empirical evidence about how developers actually use LLMs’ assistance in real-world practices remains limited. To bridge this gap, we conducted a large-scale empirical analysis of ChatGPT usage on GitHub, and we presented DevChat, a curated dataset of 2,547 publicly shared ChatGPT conversation links collected from GitHub between May 2023 and June 2024. Through comprehensively analyzing DevChat, we explored the characteristics of developer–ChatGPT interaction patterns and identified five key categories of developers’ purposes of using developer-ChatGPT conversations during software development. Additionally, we investigated the dominant development-related activities in which ChatGPT is used, and presented a mapping framework that links GitHub data sources, development-related activities, and SE tasks. The findings show that interactions are typically short and task-focused (most are 1-3 turns); developers share conversations mainly to delegate tasks, resolve problems, and acquire knowledge, revealing five purpose categories; ChatGPT is most frequently engaged for Software Implementation and Maintenance & Evolution ; we identified 39 fine-grained SE tasks supported by ChatGPT, with Code Generation & Completion as well as Code modification & Optimization being the most prominent. Our study offers a comprehensive mapping of ChatGPT’s applications in real-world software development scenarios and provides a foundation for understanding LLMs’ practical roles in software development.
- Open Access
- 15 December 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (8) , 1-38
Using LLMs in Generating Design Rationale for Software Architecture Decisions
Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Besides, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.
- 15 December 2025
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 35 (8) , 1-38
Open AccessUsing LLMs in Generating Design Rationale for Software Architecture Decisions
Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Besides, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.
- Open Access
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 530-534
Knowledge-Based Multi-Agent Framework for Automated Software Architecture Design
- 23 June 2025
- Published by Association for Computing Machinery (ACM)
- p. 530-534
Open AccessKnowledge-Based Multi-Agent Framework for Automated Software Architecture Design
- 21 November 2023
- Published by University of Groningen Press
Understanding, Analysis, and Handling of Software Architecture Erosion
- 21 November 2023
- Published by University of Groningen Press
Understanding, Analysis, and Handling of Software Architecture Erosion
- 8 August 2023
- Published by Elsevier in Information and Software Technology
- Vol. 164
Warnings: Violation symptoms indicating architecture erosion
- 8 August 2023
- Published by Elsevier in Information and Software Technology
- Vol. 164
Warnings: Violation symptoms indicating architecture erosion
- Open Access
Warnings: Violation Symptoms Indicating Architecture Erosion
As a software system evolves, its architecture tends to degrade, and gradually impedes software maintenance and evolution activities and negatively impacts the quality attributes of the system. The main root cause behind architecture erosion phenomenon derives from violation symptoms (such as violations of architecture pattern). Previous studies focus on detecting violations in software systems using architecture conformance checking approaches. However, code review comments are also rich sources that may contain extensive discussions regarding architecture violations. In this work, we investigated the characteristics of architecture violation symptoms in code review comments from the developers' perspective. We employed a set of keywords related to violation symptoms to collect 606 (out of 21,583) code review comments from four popular OSS projects in the OpenStack and Qt communities. We manually analyzed the collected 606 review comments to provide the categories and linguistic patterns of violation symptoms, as well as the reactions how developers addressed them. Our findings show that: (1) 10 categories of violation symptoms are discussed by developers during the code review process; (2) The frequently-used terms of expressing violation symptoms are "inconsistent" and "violate", and the most frequently-used linguistic pattern is Problem Discovery; (3) Refactoring and removing code are the major measures (90%) to tackle violation symptoms, while a few violation symptoms were ignored by developers. Our findings suggest that the investigation of violation symptoms can help researchers better understand the characteristics of architecture erosion and facilitate the development and maintenance activities, and developers should explicitly manage violation symptoms, not only for addressing the existing architecture violations but also preventing future violations.
Open AccessWarnings: Violation Symptoms Indicating Architecture Erosion
As a software system evolves, its architecture tends to degrade, and gradually impedes software maintenance and evolution activities and negatively impacts the quality attributes of the system. The main root cause behind architecture erosion phenomenon derives from violation symptoms (such as violations of architecture pattern). Previous studies focus on detecting violations in software systems using architecture conformance checking approaches. However, code review comments are also rich sources that may contain extensive discussions regarding architecture violations. In this work, we investigated the characteristics of architecture violation symptoms in code review comments from the developers' perspective. We employed a set of keywords related to violation symptoms to collect 606 (out of 21,583) code review comments from four popular OSS projects in the OpenStack and Qt communities. We manually analyzed the collected 606 review comments to provide the categories and linguistic patterns of violation symptoms, as well as the reactions how developers addressed them. Our findings show that: (1) 10 categories of violation symptoms are discussed by developers during the code review process; (2) The frequently-used terms of expressing violation symptoms are "inconsistent" and "violate", and the most frequently-used linguistic pattern is Problem Discovery; (3) Refactoring and removing code are the major measures (90%) to tackle violation symptoms, while a few violation symptoms were ignored by developers. Our findings suggest that the investigation of violation symptoms can help researchers better understand the characteristics of architecture erosion and facilitate the development and maintenance activities, and developers should explicitly manage violation symptoms, not only for addressing the existing architecture violations but also preventing future violations.
- Open Access
- 14 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 42-51
Code Reviewer Recommendation for Architecture Violations: An Exploratory Study
- 14 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 42-51
Open AccessCode Reviewer Recommendation for Architecture Violations: An Exploratory Study