Sara Pérez-Soler
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Sara Pérez-Soler has a total of 22 co-authors. Here are the top co-authors by number of shared publications:
- Esther Guerra
Department of Computer Science, Universidad Autónoma de Madrid, Madrid, Spain
11 shared publications - Juan de Lara
Department of Computer Science, Universidad Autónoma de Madrid, Madrid, Spain
10 shared publications Juan de Lara
Universidad Autónoma de Madrid, Madrid, Spain
5 shared publications- Pablo C. Canizares
Computer Science Department, Universidad Autónoma de Madrid, Spain
5 shared publications Esther Guerra
Universidad Autnóma de Madrid
3 shared publications- Daniel Avila
University Autnoma of Madrid (UAM), Madrid, Spain
2 shared publications
Latest Publications
- Open Access
- 25 August 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 231
A model-based solution for automated (Re-)engineering of task-oriented chatbots
- 25 August 2025
- Published by Elsevier in Journal of Systems and Software
- Vol. 231
Open AccessA model-based solution for automated (Re-)engineering of task-oriented chatbots
- Open Access
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Automating the Development of Task-oriented LLM-based Chatbots
Task-oriented chatbots are increasingly used to access all sorts of services – like booking a flight, or setting a medical appointment – through natural language conversation. There are many technologies for implementing task-oriented chatbots, including Dialogflow, Watson, and Rasa. They rely on an explicit definition of the user intents, conversation flows, and chatbot outputs, which is costly to specify, and sometimes results in suboptimal user experiences and artificial conversations with limited diversity of chatbot responses. Recently, the advances in generative artificial intelligence fostered by Large Language Models (LLMs) have enabled a new range of open-domain chatbots, like ChatGPT, able to converse fluently on any topic. However, they are general-purpose, and therefore not directly usable to solve specialised tasks reliably. In this paper, we study the power of LLMs to build task-oriented chatbots, resulting in lighter specifications – no intent definition required – and more natural conversations than in intent-based approaches. To this end, we propose a lightweight domain-specific language based on YAML to specify chatbots using modules of different types (e.g., menus, question-answering, data gathering). These specifications are compiled into structured LLM prompts that use the ReAct framework to inform our runtime how to interpret the user input and coordinate the tasks that the chatbot must perform. The paper presents the design and realisation of our framework, and an assessment that encodes a set of existing intent-based chatbots using our approach, showing its benefits in terms of specification size, conversation flexibility and output diversity.
- 8 July 2024
- Published by Association for Computing Machinery (ACM)
- p. 1-10
Open AccessAutomating the Development of Task-oriented LLM-based Chatbots
Task-oriented chatbots are increasingly used to access all sorts of services – like booking a flight, or setting a medical appointment – through natural language conversation. There are many technologies for implementing task-oriented chatbots, including Dialogflow, Watson, and Rasa. They rely on an explicit definition of the user intents, conversation flows, and chatbot outputs, which is costly to specify, and sometimes results in suboptimal user experiences and artificial conversations with limited diversity of chatbot responses. Recently, the advances in generative artificial intelligence fostered by Large Language Models (LLMs) have enabled a new range of open-domain chatbots, like ChatGPT, able to converse fluently on any topic. However, they are general-purpose, and therefore not directly usable to solve specialised tasks reliably. In this paper, we study the power of LLMs to build task-oriented chatbots, resulting in lighter specifications – no intent definition required – and more natural conversations than in intent-based approaches. To this end, we propose a lightweight domain-specific language based on YAML to specify chatbots using modules of different types (e.g., menus, question-answering, data gathering). These specifications are compiled into structured LLM prompts that use the ReAct framework to inform our runtime how to interpret the user input and coordinate the tasks that the chatbot must perform. The paper presents the design and realisation of our framework, and an assessment that encodes a set of existing intent-based chatbots using our approach, showing its benefits in terms of specification size, conversation flexibility and output diversity.
- Open Access
- 18 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 232-241
Mutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots – like ChatGPT – can converse on any topic, task-oriented chatbots – the focus of this paper – are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots. To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
- 18 June 2024
- Published by Association for Computing Machinery (ACM)
- p. 232-241
Open AccessMutation Testing for Task-Oriented Chatbots
Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots – like ChatGPT – can converse on any topic, task-oriented chatbots – the focus of this paper – are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots. To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.
- Open Access
- 17 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Software
- Vol. 41 (5) , 60-69
Integrating Static Quality Assurance in CI Chatbot Development Workflows
Chatbots are an emerging, disruptive technology to access software services via natural language conversation. Companies are adopting task-oriented chatbots to offer customer support, widen availability, lower the entry barrier to their services (e.g., for shopping, booking, or banking), and keep users engaged. This interest in chatbots has promoted the appearance of many chatbot development platforms and tools, like Google’s Dialogflow, Amazon’s Lex, or Rasa. However, there are hardly any proposals to integrate automated quality assurance mechanisms – including metrics and static analysis – into the chatbot development workflow. To fill this gap, we present a continuous integration workflow for chatbot development, implemented as GitHub actions, and show its usefulness by its application to open-source chatbots.
- 17 May 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Software
- Vol. 41 (5) , 60-69
Open AccessIntegrating Static Quality Assurance in CI Chatbot Development Workflows
Chatbots are an emerging, disruptive technology to access software services via natural language conversation. Companies are adopting task-oriented chatbots to offer customer support, widen availability, lower the entry barrier to their services (e.g., for shopping, booking, or banking), and keep users engaged. This interest in chatbots has promoted the appearance of many chatbot development platforms and tools, like Google’s Dialogflow, Amazon’s Lex, or Rasa. However, there are hardly any proposals to integrate automated quality assurance mechanisms – including metrics and static analysis – into the chatbot development workflow. To fill this gap, we present a continuous integration workflow for chatbot development, implemented as GitHub actions, and show its usefulness by its application to open-source chatbots.
- Open Access
- 17 April 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 33 (4) , 1-43
Measuring and Clustering Heterogeneous Chatbot Designs
Conversational agents, or chatbots, have become popular to access all kind of software services. They provide an intuitive natural language interface for interaction, available from a wide range of channels including social networks, web pages, intelligent speakers or cars. In response to this demand, many chatbot development platforms and tools have emerged. However, they typically lack support to statically measure properties of the chatbots being built, as indicators of their size, complexity, quality or usability. Similarly, there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies. To overcome this limitation, we propose a suite of 21 metrics for chatbot designs, as well as two clustering methods that help in grouping chatbots along their conversation topics and design features. Both the metrics and the clustering methods are defined on a neutral chatbot design language, becoming independent of the implementation platform. We provide automatic translations of chatbots defined on some major platforms into this neutral notation to perform the measurement and clustering. The approach is supported by our tool Asymob , which we have used to evaluate the metrics and the clustering methods over a set of 259 Dialogflow and Rasa chatbots from open-source repositories. The results open the door to incorporating the metrics within chatbot development processes for the early detection of quality issues, and to exploit clustering to organise large collections of chatbots into significant groups to ease chatbot comprehension, search and comparison.
- 17 April 2024
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Software Engineering and Methodology
- Vol. 33 (4) , 1-43
Open AccessMeasuring and Clustering Heterogeneous Chatbot Designs
Conversational agents, or chatbots, have become popular to access all kind of software services. They provide an intuitive natural language interface for interaction, available from a wide range of channels including social networks, web pages, intelligent speakers or cars. In response to this demand, many chatbot development platforms and tools have emerged. However, they typically lack support to statically measure properties of the chatbots being built, as indicators of their size, complexity, quality or usability. Similarly, there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies. To overcome this limitation, we propose a suite of 21 metrics for chatbot designs, as well as two clustering methods that help in grouping chatbots along their conversation topics and design features. Both the metrics and the clustering methods are defined on a neutral chatbot design language, becoming independent of the implementation platform. We provide automatic translations of chatbots defined on some major platforms into this neutral notation to perform the measurement and clustering. The approach is supported by our tool Asymob , which we have used to evaluate the metrics and the clustering methods over a set of 259 Dialogflow and Rasa chatbots from open-source repositories. The results open the door to incorporating the metrics within chatbot development processes for the early detection of quality issues, and to exploit clustering to organise large collections of chatbots into significant groups to ease chatbot comprehension, search and comparison.
- Open Access
- 15 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 23-33
Coverage-based Strategies for the Automated Synthesis of Test Scenarios for Conversational Agents
Conversational agents - or chatbots - are increasingly used as the user interface to many software services. While open-domain chatbots like ChatGPT excel in their ability to chat about any topic, task-oriented conversational agents are designed to perform goal-oriented tasks (e.g., booking or shopping) guided by a dialogue-based user interaction, which is explicitly designed. Like any kind of software system, task-oriented conversational agents need to be properly tested to ensure their quality. For this purpose, some tools permit defining and executing conversation test cases. However, there are currently no established means to assess the coverage of the design of a task-oriented agent by a test suite, or mechanisms to automate quality test case generation ensuring the agent coverage. To attack this problem, we propose test coverage criteria for task-oriented conversational agents, and define coverage-based strategies to synthesise test scenarios, some oriented to test case reduction. We provide an implementation of the criteria and the strategies that is independent of the agent development platform. Finally, we report on their evaluation on open-source Dialogflow and Rasa agents, and a comparison against a state-of-the-art testing tool. The experiment shows benefits in terms of test generation correctness, increased coverage and reduced testing time.
- 15 April 2024
- Published by Association for Computing Machinery (ACM)
- p. 23-33
Open AccessCoverage-based Strategies for the Automated Synthesis of Test Scenarios for Conversational Agents
Conversational agents - or chatbots - are increasingly used as the user interface to many software services. While open-domain chatbots like ChatGPT excel in their ability to chat about any topic, task-oriented conversational agents are designed to perform goal-oriented tasks (e.g., booking or shopping) guided by a dialogue-based user interaction, which is explicitly designed. Like any kind of software system, task-oriented conversational agents need to be properly tested to ensure their quality. For this purpose, some tools permit defining and executing conversation test cases. However, there are currently no established means to assess the coverage of the design of a task-oriented agent by a test suite, or mechanisms to automate quality test case generation ensuring the agent coverage. To attack this problem, we propose test coverage criteria for task-oriented conversational agents, and define coverage-based strategies to synthesise test scenarios, some oriented to test case reduction. We provide an implementation of the criteria and the strategies that is independent of the agent development platform. Finally, we report on their evaluation on open-source Dialogflow and Rasa agents, and a comparison against a state-of-the-art testing tool. The experiment shows benefits in terms of test generation correctness, increased coverage and reduced testing time.
- Open Access
- 12 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 130542-130562
Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings
After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.
- 12 December 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 10, 130542-130562
Open AccessUsing the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings
After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.
- Open Access
- 1 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16-20
Asymob: a platform for measuring and clustering chatbots
Chatbots have become a popular way to access all sorts of services via natural language. Many platforms and tools have been proposed for their construction, like Google’s Dialogflow, Amazon’s Lex or Rasa. However, most of them still miss integrated quality assurance methods like metrics. Moreover, there is currently a lack of mechanisms to compare and classify chatbots possibly developed with heterogeneous technologies.To tackle these issues, we present Asymob, a web platform that enables the measurement of chatbots using a suite of 20 metrics. The tool features a repository supporting chatbots built with different technologies, like Dialogflow and Rasa. Asymob’s metrics help in detecting quality issues and serve to compare chatbots across and within technologies. The tool also helps in classifying chatbots along conversation topics or design features by means of two clustering methods: based on the chatbot metrics or on the phrases expected and produced by the chatbot. A video showcasing the tool is available at https://www.youtube.com/watch?v=8lpETkILpv8.
- 1 May 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 16-20
Open AccessAsymob: a platform for measuring and clustering chatbots
Chatbots have become a popular way to access all sorts of services via natural language. Many platforms and tools have been proposed for their construction, like Google’s Dialogflow, Amazon’s Lex or Rasa. However, most of them still miss integrated quality assurance methods like metrics. Moreover, there is currently a lack of mechanisms to compare and classify chatbots possibly developed with heterogeneous technologies.To tackle these issues, we present Asymob, a web platform that enables the measurement of chatbots using a suite of 20 metrics. The tool features a repository supporting chatbots built with different technologies, like Dialogflow and Rasa. Asymob’s metrics help in detecting quality issues and serve to compare chatbots across and within technologies. The tool also helps in classifying chatbots along conversation topics or design features by means of two clustering methods: based on the chatbot metrics or on the phrases expected and produced by the chatbot. A video showcasing the tool is available at https://www.youtube.com/watch?v=8lpETkILpv8.
- 26 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Software
- Vol. 38 (4) , 94-103
Choosing a Chatbot Development Tool
Chatbots are programs that supply services to users via conversation in natural language, acting as virtual assistants within social networks or web applications. Here, we review the most representative chatbot development tools with a focus on technical and managerial aspects.
- 26 February 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Software
- Vol. 38 (4) , 94-103
Choosing a Chatbot Development Tool
Chatbots are programs that supply services to users via conversation in natural language, acting as virtual assistants within social networks or web applications. Here, we review the most representative chatbot development tools with a focus on technical and managerial aspects.