Kyller Gorgônio
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Kyller Gorgônio has a total of 51 co-authors. Here are the top co-authors by number of shared publications:
- Angelo Perkusich
UFCG, Campina Grande, Brazil
24 shared publications - Danyllo Albuquerque
IFPB
16 shared publications - Mirko Perkusich
UFCG
16 shared publications - Hyggo Almeida
VIRTUS/UFCG, Campina Grande, Brazil
9 shared publications - Felipe Cunha
UFCG
6 shared publications - Dalton Cézane Gomes Valadares
Federal University of Paraba, Bananeiras, Brazil
5 shared publications
Latest Publications
- Open Access
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 9-16
Integrating Bayesian Reasoning and Evolutionary Search for Knowledge-Driven Team Formation
[Context] Forming software project teams requires balancing technical requirements, prior collaboration, and organizational constraints. Genetic Algorithms (GAs) can explore this combinatorial space effectively, but their fitness functions are often researcher-defined, misaligned with real decision criteria, and difficult to audit. [Objective] We investigate whether expert-elicited, symbolic–probabilistic knowledge can be integrated into GA-based search to support transparent and traceable single-project team formation. [Method] Following a Design Science approach with an industrial partner, we elicited how a senior staffing authority evaluates candidate teams and formalized this reasoning as a compact Bayesian Network (BN). The BN integrates rule-based assessments of technical coverage and graph-based measures of prior collaboration. During optimization, the BN is queried to evaluate candidate teams, and its approval probability directly serves as the GA’s fitness score. [Results] An expert workshop involving six representative project scenarios shows that the BN’s team rankings are broadly aligned with the staffing authority’s judgments, providing initial evidence that the encoded approval model reflects real-world decision preferences. In addition, runtime measurements of full GA executions using a developer pool of 192 profiles indicate that the BN-based evaluator is computationally feasible for offline or batch team formation scenarios. [Conclusion] These results suggest that expert-calibrated, symbolic–probabilistic evaluators can be operationalized as fitness functions in evolutionary search, enabling knowledge-driven team formation aligned with expert reasoning, with an emphasis on interpretability, auditability, and feasibility.
- 12 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 9-16
Open AccessIntegrating Bayesian Reasoning and Evolutionary Search for Knowledge-Driven Team Formation
[Context] Forming software project teams requires balancing technical requirements, prior collaboration, and organizational constraints. Genetic Algorithms (GAs) can explore this combinatorial space effectively, but their fitness functions are often researcher-defined, misaligned with real decision criteria, and difficult to audit. [Objective] We investigate whether expert-elicited, symbolic–probabilistic knowledge can be integrated into GA-based search to support transparent and traceable single-project team formation. [Method] Following a Design Science approach with an industrial partner, we elicited how a senior staffing authority evaluates candidate teams and formalized this reasoning as a compact Bayesian Network (BN). The BN integrates rule-based assessments of technical coverage and graph-based measures of prior collaboration. During optimization, the BN is queried to evaluate candidate teams, and its approval probability directly serves as the GA’s fitness score. [Results] An expert workshop involving six representative project scenarios shows that the BN’s team rankings are broadly aligned with the staffing authority’s judgments, providing initial evidence that the encoded approval model reflects real-world decision preferences. In addition, runtime measurements of full GA executions using a developer pool of 192 profiles indicate that the BN-based evaluator is computationally feasible for offline or batch team formation scenarios. [Conclusion] These results suggest that expert-calibrated, symbolic–probabilistic evaluators can be operationalized as fitness functions in evolutionary search, enabling knowledge-driven team formation aligned with expert reasoning, with an emphasis on interpretability, auditability, and feasibility.
- Open Access
- 1 April 2026
- Published by Springer Nature
- p. 155-174
Evaluating the Quality of User Stories: An Extended Comparative Study of Multiple LLMs and Rule-Based Tools
Background: Ensuring the quality of user stories is vital to Agile Software Development. Rule-based tools like AQUSA, based on the Quality User Story (QUS) framework, offer reliable structural checks but struggle with context-sensitive or pragmatic issues. Large Language Models (LLMs) have emerged as potential alternatives, yet prior studies often rely on small datasets, older models, or lack direct comparison with rule-based baselines. Objective: This study aims to assess the effectiveness of modern LLMs relative to a rule-based tool (AQUSA) for detecting defects in user stories, considering both structural and contextual dimensions. Method: We conduct a large-scale comparative evaluation involving AQUSA and three GPT-family LLMs (GPT-5, GPT-5-mini, and GPT-4), using 182 user stories drawn from three industrial datasets. We apply both quantitative metrics (precision, recall, F1-score) and qualitative analysis of feedback clarity and defect relevance. Results: GPT-5-mini achieved the highest recall (0.81) and overall F1-score (0.62), while AQUSA attained the highest precision (0.61) with significantly fewer false positives. GPT-5 showed high hallucination rates and instability; GPT-4 was overly conservative, leading to under-detection of defects. Conclusion: Neither rule-based nor GPT-family LLM-based approaches suffice in isolation. Rule-based tools enforce structural rigor, while LLMs capture nuanced linguistic and pragmatic flaws. We advocate a hybrid “Dual-gate” strategy—using AQUSA for structural validation followed by lightweight LLMs for contextual refinement—to improve the reliability and scalability of user story quality assessment in agile environments.
- 1 April 2026
- Published by Springer Nature
- p. 155-174
Open AccessEvaluating the Quality of User Stories: An Extended Comparative Study of Multiple LLMs and Rule-Based Tools
Background: Ensuring the quality of user stories is vital to Agile Software Development. Rule-based tools like AQUSA, based on the Quality User Story (QUS) framework, offer reliable structural checks but struggle with context-sensitive or pragmatic issues. Large Language Models (LLMs) have emerged as potential alternatives, yet prior studies often rely on small datasets, older models, or lack direct comparison with rule-based baselines. Objective: This study aims to assess the effectiveness of modern LLMs relative to a rule-based tool (AQUSA) for detecting defects in user stories, considering both structural and contextual dimensions. Method: We conduct a large-scale comparative evaluation involving AQUSA and three GPT-family LLMs (GPT-5, GPT-5-mini, and GPT-4), using 182 user stories drawn from three industrial datasets. We apply both quantitative metrics (precision, recall, F1-score) and qualitative analysis of feedback clarity and defect relevance. Results: GPT-5-mini achieved the highest recall (0.81) and overall F1-score (0.62), while AQUSA attained the highest precision (0.61) with significantly fewer false positives. GPT-5 showed high hallucination rates and instability; GPT-4 was overly conservative, leading to under-detection of defects. Conclusion: Neither rule-based nor GPT-family LLM-based approaches suffice in isolation. Rule-based tools enforce structural rigor, while LLMs capture nuanced linguistic and pragmatic flaws. We advocate a hybrid “Dual-gate” strategy—using AQUSA for structural validation followed by lightweight LLMs for contextual refinement—to improve the reliability and scalability of user story quality assessment in agile environments.
- Open Access
- 31 March 2026
- Published by Sociedade Brasileira de Computação in Journal of Software Engineering Research and Development
- Vol. 14 (1) , 517-536
From Textual Requirements to Microservice Architectures: A Comprehensive Evaluation of LLM-Based Design Synthesis
[Context]. Microservice architectures have become a dominant paradigm for modernizing monolithic systems. However, identifying appropriate services remains a challenging and largely manual task. Existing decomposition approaches are predominantly code-centric, limiting their applicability in early design stages where only textual requirements are available. [Problem]. Despite recent advances in Large Language Models (LLMs), there is still limited empirical evidence regarding their ability to synthesize complete microservice architectures directly from natural-language requirements, including both service definitions and inter-service interactions. [Goal]. This study investigates whether an LLM can bridge the gap between requirements engineering and architectural design by generating microservice architectures solely from textual requirements, and evaluates the structural agreement and perceived quality of the generated solutions. [Method]. We conduct a mixed-method study using the OpenAI o3 model under zero-shot (ZS) and few-shot (FS) prompting strategies across two systems (Bookstore and PetClinic), with one execution per system and prompting condition. Generated architectures are evaluated through (i) quantitative comparison with implemented reference architectures using precision, recall, and F1-score for service identification and communication recovery, and (ii) a blinded expert assessment of correctness, completeness, modularity, and plausibility, supplemented by a structured descriptive synthesis of open-ended feedback. [Results]. The findings indicate that OpenAI o3 can identify services from requirements, achieving higher agreement with the references under FS prompting (F1 ≈ 0.79 for ZS and ≈ 0.97 for FS). Communication recovery is more challenging, with ZS producing overly dense architectures characterized by high recall but low precision (F1 ≈ 0.61). FS prompting improves structural agreement and perceived architectural quality, achieving F1 ≈ 0.82 for communication recovery while reducing unsupported dependencies. Expert evaluation corroborates these findings, indicating that FS-generated architectures are consistently perceived as more modular, coherent, and plausible than ZS outputs. [Conclusion]. Within the evaluated scope, OpenAI o3 shows potential to support requirements-driven architectural synthesis, particularly when guided by minimal exemplar-based prompting. The results should be interpreted as model- and context-specific evidence from two relatively small systems and a single execution per condition, rather than as model-independent proof of effectiveness.
- 31 March 2026
- Published by Sociedade Brasileira de Computação in Journal of Software Engineering Research and Development
- Vol. 14 (1) , 517-536
Open AccessFrom Textual Requirements to Microservice Architectures: A Comprehensive Evaluation of LLM-Based Design Synthesis
[Context]. Microservice architectures have become a dominant paradigm for modernizing monolithic systems. However, identifying appropriate services remains a challenging and largely manual task. Existing decomposition approaches are predominantly code-centric, limiting their applicability in early design stages where only textual requirements are available. [Problem]. Despite recent advances in Large Language Models (LLMs), there is still limited empirical evidence regarding their ability to synthesize complete microservice architectures directly from natural-language requirements, including both service definitions and inter-service interactions. [Goal]. This study investigates whether an LLM can bridge the gap between requirements engineering and architectural design by generating microservice architectures solely from textual requirements, and evaluates the structural agreement and perceived quality of the generated solutions. [Method]. We conduct a mixed-method study using the OpenAI o3 model under zero-shot (ZS) and few-shot (FS) prompting strategies across two systems (Bookstore and PetClinic), with one execution per system and prompting condition. Generated architectures are evaluated through (i) quantitative comparison with implemented reference architectures using precision, recall, and F1-score for service identification and communication recovery, and (ii) a blinded expert assessment of correctness, completeness, modularity, and plausibility, supplemented by a structured descriptive synthesis of open-ended feedback. [Results]. The findings indicate that OpenAI o3 can identify services from requirements, achieving higher agreement with the references under FS prompting (F1 ≈ 0.79 for ZS and ≈ 0.97 for FS). Communication recovery is more challenging, with ZS producing overly dense architectures characterized by high recall but low precision (F1 ≈ 0.61). FS prompting improves structural agreement and perceived architectural quality, achieving F1 ≈ 0.82 for communication recovery while reducing unsupported dependencies. Expert evaluation corroborates these findings, indicating that FS-generated architectures are consistently perceived as more modular, coherent, and plausible than ZS outputs. [Conclusion]. Within the evaluated scope, OpenAI o3 shows potential to support requirements-driven architectural synthesis, particularly when guided by minimal exemplar-based prompting. The results should be interpreted as model- and context-specific evidence from two relatively small systems and a single execution per condition, rather than as model-independent proof of effectiveness.
- Open Access
- 31 March 2026
- Published by Sociedade Brasileira de Computação in Journal of Software Engineering Research and Development
- Vol. 14 (1) , 460-481
An Expert-Guided Method for Developer Compatibility Graphs and Bayesian Team-Fit Evaluation
[Context]. Forming effective software teams remains challenging, particularly in large organizations managing multiple concurrent projects. Existing approaches to modeling developer compatibility often rely on subjective personality traits, purely structural proxies, or large volumes of historical interaction data, limiting both their interpretability and their alignment with how staffing decisions are made in practice. [Objective]. This paper proposes a lightweight, expert-guided method for modeling developer compatibility and operationalizing collaboration evidence for transparent and explainable team-fit evaluation. [Method]. We introduce a regression-calibrated Pair Compatibility (PC) model that combines two project-level indicators, Objective Success Factor (OSF) and Subjective Leadership Factor (SLF), with an expert-defined saturation effect for repeated collaborations. During multiple expert workshops, the expert validated the relevance and conceptual interpretation of OSF and SLF, their normalized scales, and their use in compatibility modeling. The model is calibrated using eight synthetic collaboration scenarios labeled by the expert and is then applied to an organizational analysis base parameterized from anonymized project-participation records and project-level OSF/SLF indicators, thereby producing a weighted developer compatibility graph. Pairwise compatibility evidence is then transformed into a size-agnostic collaboration profile and integrated into an expert-calibrated Bayesian Network (BN) that estimates overall Team Fit. [Results]. In an industrial organization with more than 300 professionals, the calibrated PC regression achieved a Mean Absolute Error (MAE) of 0.0812 and a Pearson correlation of 0.80 with expert expectations on the eight labeled scenarios. When applied to the organizational dataset of 192 developers, and after the PC > 0.3 filter, 773 valid developer pairs were retained; the resulting compatibility graph illustrates how the method produces cohesive collaboration clusters and identifies bridge developers to support managerial staffing decisions. The BN-based team evaluator produced Brier scores of 0.0788 and 0.0782 for Team Fit (AE) in scenario-based validation, reflecting the concentrated posterior distributions of the calibrated model (σ = 0.05) relative to the smoother expert-provided reference distributions. For Collaboration Fit (AC), the PC-tier distribution representation ranked collaboration quality in closer agreement with expert judgment than a mean-based encoding (Spearman ρ = 0.882 vs. ρ = 0.812). [Conclusion]. By combining expert-validated indicator definitions, organizational co-participation records, project-level OSF/SLF indicators, and a small set of synthetic expert-labeled scenarios, the proposed method yields an interpretable compatibility graph and a transparent probabilistic evaluator that bridges pairwise collaboration evidence and team-level decision-making. The results support the feasibility of expert-guided regression and Bayesian reasoning as a practical and explainable alternative to data-intensive black-box models for software team formation (STF), particularly in settings where interpretability and alignment with local staffing authority are central requirements.
- 31 March 2026
- Published by Sociedade Brasileira de Computação in Journal of Software Engineering Research and Development
- Vol. 14 (1) , 460-481
Open AccessAn Expert-Guided Method for Developer Compatibility Graphs and Bayesian Team-Fit Evaluation
[Context]. Forming effective software teams remains challenging, particularly in large organizations managing multiple concurrent projects. Existing approaches to modeling developer compatibility often rely on subjective personality traits, purely structural proxies, or large volumes of historical interaction data, limiting both their interpretability and their alignment with how staffing decisions are made in practice. [Objective]. This paper proposes a lightweight, expert-guided method for modeling developer compatibility and operationalizing collaboration evidence for transparent and explainable team-fit evaluation. [Method]. We introduce a regression-calibrated Pair Compatibility (PC) model that combines two project-level indicators, Objective Success Factor (OSF) and Subjective Leadership Factor (SLF), with an expert-defined saturation effect for repeated collaborations. During multiple expert workshops, the expert validated the relevance and conceptual interpretation of OSF and SLF, their normalized scales, and their use in compatibility modeling. The model is calibrated using eight synthetic collaboration scenarios labeled by the expert and is then applied to an organizational analysis base parameterized from anonymized project-participation records and project-level OSF/SLF indicators, thereby producing a weighted developer compatibility graph. Pairwise compatibility evidence is then transformed into a size-agnostic collaboration profile and integrated into an expert-calibrated Bayesian Network (BN) that estimates overall Team Fit. [Results]. In an industrial organization with more than 300 professionals, the calibrated PC regression achieved a Mean Absolute Error (MAE) of 0.0812 and a Pearson correlation of 0.80 with expert expectations on the eight labeled scenarios. When applied to the organizational dataset of 192 developers, and after the PC > 0.3 filter, 773 valid developer pairs were retained; the resulting compatibility graph illustrates how the method produces cohesive collaboration clusters and identifies bridge developers to support managerial staffing decisions. The BN-based team evaluator produced Brier scores of 0.0788 and 0.0782 for Team Fit (AE) in scenario-based validation, reflecting the concentrated posterior distributions of the calibrated model (σ = 0.05) relative to the smoother expert-provided reference distributions. For Collaboration Fit (AC), the PC-tier distribution representation ranked collaboration quality in closer agreement with expert judgment than a mean-based encoding (Spearman ρ = 0.882 vs. ρ = 0.812). [Conclusion]. By combining expert-validated indicator definitions, organizational co-participation records, project-level OSF/SLF indicators, and a small set of synthetic expert-labeled scenarios, the proposed method yields an interpretable compatibility graph and a transparent probabilistic evaluator that bridges pairwise collaboration evidence and team-level decision-making. The results support the feasibility of expert-guided regression and Bayesian reasoning as a practical and explainable alternative to data-intensive black-box models for software team formation (STF), particularly in settings where interpretability and alignment with local staffing authority are central requirements.
- Open Access
- 23 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1717-1726
Towards a Semi-Automated Pipeline for User Story Curation and Classification
- 23 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1717-1726
Open AccessTowards a Semi-Automated Pipeline for User Story Curation and Classification
- Open Access
- 23 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1700-1707
Do LLMs Agree on Microservice Decompositions? A Multi–Model Study from Textual Requirements
- 23 March 2026
- Published by Association for Computing Machinery (ACM)
- p. 1700-1707
Open AccessDo LLMs Agree on Microservice Decompositions? A Multi–Model Study from Textual Requirements
- Open Access
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 19-24
A Bayesian Network for Evaluating Software Project Team Fit: An Industrial Case Study
[Context] Forming effective software development teams requires balancing both technical and collaborative factors. While optimization methods—such as genetic algorithms—have been applied to automate team formation, they typically rely on fitness functions built from fixed weights or handcrafted rules. These ad hoc formulations rarely reflect structured expert reasoning and may misalign with organizational expectations. [Objective] This study focuses on constructing a Bayesian Network (BN) that formally encodes expert knowledge about team-project fit. The BN is intended to serve as a reusable, interpretable fitness function in future optimizationbased team formation systems. [Method] We conducted a case study in a large software organization with over 300 employees. Through expert workshops, we identified relevant developer capabilities, defined qualitative states, and structured the BN accordingly. Conditional Probability Tables (CPTs) were populated using expertlabeled what-if scenarios, then calibrated using the Ranked Nodes method. [Results] The final BN comprises nodes for technological and collaboration fit. Validation against held-out, expert-labeled scenarios yielded consistently low Brier Scores, demonstrating strong alignment with expert expectations. [Conclusion] BNs offer a viable, interpretable mechanism for encoding expert knowledge in team formation. This work establishes a structured and transparent foundation for future integration into automated team formation workflows.
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 19-24
Open AccessA Bayesian Network for Evaluating Software Project Team Fit: An Industrial Case Study
[Context] Forming effective software development teams requires balancing both technical and collaborative factors. While optimization methods—such as genetic algorithms—have been applied to automate team formation, they typically rely on fitness functions built from fixed weights or handcrafted rules. These ad hoc formulations rarely reflect structured expert reasoning and may misalign with organizational expectations. [Objective] This study focuses on constructing a Bayesian Network (BN) that formally encodes expert knowledge about team-project fit. The BN is intended to serve as a reusable, interpretable fitness function in future optimizationbased team formation systems. [Method] We conducted a case study in a large software organization with over 300 employees. Through expert workshops, we identified relevant developer capabilities, defined qualitative states, and structured the BN accordingly. Conditional Probability Tables (CPTs) were populated using expertlabeled what-if scenarios, then calibrated using the Ranked Nodes method. [Results] The final BN comprises nodes for technological and collaboration fit. Validation against held-out, expert-labeled scenarios yielded consistently low Brier Scores, demonstrating strong alignment with expert expectations. [Conclusion] BNs offer a viable, interpretable mechanism for encoding expert knowledge in team formation. This work establishes a structured and transparent foundation for future integration into automated team formation workflows.
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 31-36
Constructing Developer Compatibility Graphs for Team Formation: An Industrial Case Study
[Context]. Forming effective software teams remains difficult, especially in large organizations that juggle many parallel projects; existing compatibility models either rely on subjective personality traits or demand extensive historical data. [Objective] We introduce a lightweight, hybrid method that fuses minimal project metrics with expert judgment to build a weighted compatibility graph of developers. [Method] In an industrial case study (300+ professionals), a senior technical leader labeled eight synthetic collaboration scenarios that span the space of Objective Success Factor (OSF) and Subjective Leadership Factor (SLF). Those labels calibrated an ordinary-least-squares regression that also includes a saturation term for repeated pairings; the model was then applied to the organization’s project history. [Results] The regression achieved a Mean Absolute Error (MAE) of 0.11 and a Pearson correlation of 0.86 against the expert ground truth. When rolled out to real data (62 developers, 147 valid pairs), it produced PC weights ranging from 0.12 to 0.93, uncovered four cohesive clusters, and highlighted several bridge developers—information that managers subsequently used to re-balance squads. [Conclusion] By combining three readily available project metrics with a small set of expert-labeled scenarios, our method yields a transparent compatibility graph that matches the utility of data-hungry black-box models. It equips managers with an out-of-the-box aid for data-driven team design and provides researchers with a replicable blueprint for analyzing collaboration in data-constrained environments.
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 31-36
Constructing Developer Compatibility Graphs for Team Formation: An Industrial Case Study
[Context]. Forming effective software teams remains difficult, especially in large organizations that juggle many parallel projects; existing compatibility models either rely on subjective personality traits or demand extensive historical data. [Objective] We introduce a lightweight, hybrid method that fuses minimal project metrics with expert judgment to build a weighted compatibility graph of developers. [Method] In an industrial case study (300+ professionals), a senior technical leader labeled eight synthetic collaboration scenarios that span the space of Objective Success Factor (OSF) and Subjective Leadership Factor (SLF). Those labels calibrated an ordinary-least-squares regression that also includes a saturation term for repeated pairings; the model was then applied to the organization’s project history. [Results] The regression achieved a Mean Absolute Error (MAE) of 0.11 and a Pearson correlation of 0.86 against the expert ground truth. When rolled out to real data (62 developers, 147 valid pairs), it produced PC weights ranging from 0.12 to 0.93, uncovered four cohesive clusters, and highlighted several bridge developers—information that managers subsequently used to re-balance squads. [Conclusion] By combining three readily available project metrics with a small set of expert-labeled scenarios, our method yields a transparent compatibility graph that matches the utility of data-hungry black-box models. It equips managers with an out-of-the-box aid for data-driven team design and provides researchers with a replicable blueprint for analyzing collaboration in data-constrained environments.
- Open Access
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 25-30
How Software Testers Perceive Large Language Models: A Focus Group Study with Lexicometric Analysis
Context. Large Language Models (LLMs) are increasingly embedded in software-engineering toolchains, with potential to enhance quality-assurance tasks such as test-case generation, bug analysis, and traceability. Yet their industrial value remains unclear, partly because practicing testers’ perspectives are underexplored. Objective. We investigate how industry testers perceive the usefulness and limitations of LLMs across the Software Test Life Cycle (STLC), identifying (i) the most critical stages, (ii) operational bottlenecks, and (iii) perceived benefits, risks, and adoption conditions. Method. A focus group with five professional quality analysts (2–10 years’ experience) included hands-on interaction with an LLM-powered BDD notebook, followed by guided discussion and affinity-diagram voting. Forty-two statements were coded and prioritized. Results. Requirements Analysis and Test Planning were deemed most critical; 71% of votes linked rework to unclear or incomplete requirements. Benefits cited were automation of repetitive tasks, broader coverage, and faster learning; main concerns were prompt sensitivity, limited domain generalization, and data-privacy risks, with emphasis on human oversight and domain-adapted prompt libraries. Conclusion. While LLMs can improve efficiency and coverage, adoption depends on high-quality inputs, secure deployment, and early-phase integration. The findings offer empirically grounded guidance for aligning LLM solutions with the socio-technical realities of industrial testing teams.
- 23 September 2025
- Published by Sociedade Brasileira de Computação
- p. 25-30
Open AccessHow Software Testers Perceive Large Language Models: A Focus Group Study with Lexicometric Analysis
Context. Large Language Models (LLMs) are increasingly embedded in software-engineering toolchains, with potential to enhance quality-assurance tasks such as test-case generation, bug analysis, and traceability. Yet their industrial value remains unclear, partly because practicing testers’ perspectives are underexplored. Objective. We investigate how industry testers perceive the usefulness and limitations of LLMs across the Software Test Life Cycle (STLC), identifying (i) the most critical stages, (ii) operational bottlenecks, and (iii) perceived benefits, risks, and adoption conditions. Method. A focus group with five professional quality analysts (2–10 years’ experience) included hands-on interaction with an LLM-powered BDD notebook, followed by guided discussion and affinity-diagram voting. Forty-two statements were coded and prioritized. Results. Requirements Analysis and Test Planning were deemed most critical; 71% of votes linked rework to unclear or incomplete requirements. Benefits cited were automation of repetitive tasks, broader coverage, and faster learning; main concerns were prompt sensitivity, limited domain generalization, and data-privacy risks, with emphasis on human oversight and domain-adapted prompt libraries. Conclusion. While LLMs can improve efficiency and coverage, adoption depends on high-quality inputs, secure deployment, and early-phase integration. The findings offer empirically grounded guidance for aligning LLM solutions with the socio-technical realities of industrial testing teams.
- Open Access
- 22 September 2025
- Published by Sociedade Brasileira de Computação
- p. 171-181
A Comparative Study of LLMs for Gherkin Generation
[Context] Behavior-Driven Development (BDD) is widely adopted, but the manual creation of Gherkin scenarios remains a significant bottleneck. While Large Language Models (LLMs) show promise for automation, there is a lack of empirical evidence on their accuracy and stability when converting free-form test descriptions into structured Gherkin, creating risks for industrial adoption. Manual scenario authoring is also time-consuming and prone to inconsistencies, leading to miscommunication between technical and non-technical stakeholders and impacting software quality assurance. [Objective] This study addresses this gap by investigating the use of LLMs to automate the generation of Gherkin-based BDD scenarios from real-world, free-form test case descriptions. The goal is to assess the robustness of current models when handling informal, ambiguous, and diverse inputs typically found in practice. [Method]We conducted a comparative evaluation involving seven LLMs — GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o Mini, LLaMA 3, Phi-3, Gemini, and DeepSeek R1 — using zero-shot, one-shot, and fewshot prompting strategies. The models generated BDD scenarios from a stratified sample of ten test descriptions selected from a corpus of 1,286, ensuring diversity in structure and domain complexity. We assessed quality and consistency using quantitative metrics (METEOR, variability analysis) and Repeated Measures ANOVA to test statistical significance. [Results] The analysis revealed that simple zero-shot prompting was highly effective, achieving results comparable to more complex example-based prompting. For the topperforming model, Gemini, which balanced accuracy and stability, the difference between zero-shot and few-shot was not statistically significant. Performance differences across models were often small, suggesting that practical factors like integration and cost should also guide model choice. Some models showed higher output variability, raising concerns about consistency in test generation workflows. [Conclusion] This paper offers practical insights into prompt design and model selection for LLM-based BDD scenario generation. Results show that effective zero-shot prompts can enable scalable, high-quality generation comparable to more complex techniques, simplifying LLM adoption in industrial testing. These findings suggest that LLMs can be leveraged with minimal setup to streamline BDD, reduce costs, and accelerate validation cycles.
- 22 September 2025
- Published by Sociedade Brasileira de Computação
- p. 171-181
Open AccessA Comparative Study of LLMs for Gherkin Generation
[Context] Behavior-Driven Development (BDD) is widely adopted, but the manual creation of Gherkin scenarios remains a significant bottleneck. While Large Language Models (LLMs) show promise for automation, there is a lack of empirical evidence on their accuracy and stability when converting free-form test descriptions into structured Gherkin, creating risks for industrial adoption. Manual scenario authoring is also time-consuming and prone to inconsistencies, leading to miscommunication between technical and non-technical stakeholders and impacting software quality assurance. [Objective] This study addresses this gap by investigating the use of LLMs to automate the generation of Gherkin-based BDD scenarios from real-world, free-form test case descriptions. The goal is to assess the robustness of current models when handling informal, ambiguous, and diverse inputs typically found in practice. [Method]We conducted a comparative evaluation involving seven LLMs — GPT-3.5 Turbo, GPT-4 Turbo, GPT-4o Mini, LLaMA 3, Phi-3, Gemini, and DeepSeek R1 — using zero-shot, one-shot, and fewshot prompting strategies. The models generated BDD scenarios from a stratified sample of ten test descriptions selected from a corpus of 1,286, ensuring diversity in structure and domain complexity. We assessed quality and consistency using quantitative metrics (METEOR, variability analysis) and Repeated Measures ANOVA to test statistical significance. [Results] The analysis revealed that simple zero-shot prompting was highly effective, achieving results comparable to more complex example-based prompting. For the topperforming model, Gemini, which balanced accuracy and stability, the difference between zero-shot and few-shot was not statistically significant. Performance differences across models were often small, suggesting that practical factors like integration and cost should also guide model choice. Some models showed higher output variability, raising concerns about consistency in test generation workflows. [Conclusion] This paper offers practical insights into prompt design and model selection for LLM-based BDD scenario generation. Results show that effective zero-shot prompts can enable scalable, high-quality generation comparable to more complex techniques, simplifying LLM adoption in industrial testing. These findings suggest that LLMs can be leveraged with minimal setup to streamline BDD, reduce costs, and accelerate validation cycles.