SPSara Pérez-Soler

Sara Pérez-Soler

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Sara Pérez-Soler has a total of 22 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Automating the Development of Task-oriented LLM-based Chatbots

    by Jesús Sánchez Cuadrado, Sara Pérez-Soler, Esther Guerra, Juan De Lara

    Task-oriented chatbots are increasingly used to access all sorts of services – like booking a flight, or setting a medical appointment – through natural language conversation. There are many technologies for implementing task-oriented chatbots, including Dialogflow, Watson, and Rasa. They rely on an explicit definition of the user intents, conversation flows, and chatbot outputs, which is costly to specify, and sometimes results in suboptimal user experiences and artificial conversations with limited diversity of chatbot responses. Recently, the advances in generative artificial intelligence fostered by Large Language Models (LLMs) have enabled a new range of open-domain chatbots, like ChatGPT, able to converse fluently on any topic. However, they are general-purpose, and therefore not directly usable to solve specialised tasks reliably. In this paper, we study the power of LLMs to build task-oriented chatbots, resulting in lighter specifications – no intent definition required – and more natural conversations than in intent-based approaches. To this end, we propose a lightweight domain-specific language based on YAML to specify chatbots using modules of different types (e.g., menus, question-answering, data gathering). These specifications are compiled into structured LLM prompts that use the ReAct framework to inform our runtime how to interpret the user input and coordinate the tasks that the chatbot must perform. The paper presents the design and realisation of our framework, and an assessment that encodes a set of existing intent-based chatbots using our approach, showing its benefits in terms of specification size, conversation flexibility and output diversity.


  • Open Access

    Mutation Testing for Task-Oriented Chatbots

    by Pablo Gómez-Abajo, Sara Pérez-Soler, Pablo C. Cañizares, Esther Guerra, Juan de Lara

    Conversational agents, or chatbots, are increasingly used to access all sorts of services using natural language. While open-domain chatbots – like ChatGPT – can converse on any topic, task-oriented chatbots – the focus of this paper – are designed for specific tasks, like booking a flight, obtaining customer support, or setting an appointment. Like any other software, task-oriented chatbots need to be properly tested, usually by defining and executing test scenarios (i.e., sequences of user-chatbot interactions). However, there is currently a lack of methods to quantify the completeness and strength of such test scenarios, which can lead to low-quality tests, and hence to buggy chatbots. To fill this gap, we propose adapting mutation testing (MuT) for task-oriented chatbots. To this end, we introduce a set of mutation operators that emulate faults in chatbot designs, an architecture that enables MuT on chatbots built using heterogeneous technologies, and a practical realisation as an Eclipse plugin. Moreover, we evaluate the applicability, effectiveness and efficiency of our approach on open-source chatbots, with promising results.


  • Open Access

    Integrating Static Quality Assurance in CI Chatbot Development Workflows

    by Jesús Sánchez Cuadrado, Daniel Ávila, Sara Pérez-Soler, Pablo C. Cañizares, Esther Guerra, Juan de Lara

    Chatbots are an emerging, disruptive technology to access software services via natural language conversation. Companies are adopting task-oriented chatbots to offer customer support, widen availability, lower the entry barrier to their services (e.g., for shopping, booking, or banking), and keep users engaged. This interest in chatbots has promoted the appearance of many chatbot development platforms and tools, like Google’s Dialogflow, Amazon’s Lex, or Rasa. However, there are hardly any proposals to integrate automated quality assurance mechanisms – including metrics and static analysis – into the chatbot development workflow. To fill this gap, we present a continuous integration workflow for chatbot development, implemented as GitHub actions, and show its usefulness by its application to open-source chatbots.


  • Open Access

    Measuring and Clustering Heterogeneous Chatbot Designs

    by Pablo C. Cañizares, Jose María López-Morales, Sara Pérez-Soler, Esther Guerra, Juan de Lara

    Conversational agents, or chatbots, have become popular to access all kind of software services. They provide an intuitive natural language interface for interaction, available from a wide range of channels including social networks, web pages, intelligent speakers or cars. In response to this demand, many chatbot development platforms and tools have emerged. However, they typically lack support to statically measure properties of the chatbots being built, as indicators of their size, complexity, quality or usability. Similarly, there are hardly any mechanisms to compare and cluster chatbots developed with heterogeneous technologies. To overcome this limitation, we propose a suite of 21 metrics for chatbot designs, as well as two clustering methods that help in grouping chatbots along their conversation topics and design features. Both the metrics and the clustering methods are defined on a neutral chatbot design language, becoming independent of the implementation platform. We provide automatic translations of chatbots defined on some major platforms into this neutral notation to perform the measurement and clustering. The approach is supported by our tool Asymob , which we have used to evaluate the metrics and the clustering methods over a set of 259 Dialogflow and Rasa chatbots from open-source repositories. The results open the door to incorporating the metrics within chatbot development processes for the early detection of quality issues, and to exploit clustering to organise large collections of chatbots into significant groups to ease chatbot comprehension, search and comparison.


  • Open Access

    Coverage-based Strategies for the Automated Synthesis of Test Scenarios for Conversational Agents

    by Pablo C. Canizares, Daniel Ávila, Sara Perez-Soler, Esther Guerra, Juan De Lara

    Conversational agents - or chatbots - are increasingly used as the user interface to many software services. While open-domain chatbots like ChatGPT excel in their ability to chat about any topic, task-oriented conversational agents are designed to perform goal-oriented tasks (e.g., booking or shopping) guided by a dialogue-based user interaction, which is explicitly designed. Like any kind of software system, task-oriented conversational agents need to be properly tested to ensure their quality. For this purpose, some tools permit defining and executing conversation test cases. However, there are currently no established means to assess the coverage of the design of a task-oriented agent by a test suite, or mechanisms to automate quality test case generation ensuring the agent coverage. To attack this problem, we propose test coverage criteria for task-oriented conversational agents, and define coverage-based strategies to synthesise test scenarios, some oriented to test case reduction. We provide an implementation of the criteria and the strategies that is independent of the agent development platform. Finally, we report on their evaluation on open-source Dialogflow and Rasa agents, and a comparison against a state-of-the-art testing tool. The experiment shows benefits in terms of test generation correctness, increased coverage and reduced testing time.


  • Open Access

    Using the SOCIO Chatbot for UML Modeling: A Second Family of Experiments on Usability in Academic Settings

    by Ranci Ren, Sara Pérez-Soler, John W. Castro, Oscar Dieste, Silvia T. Acuña

    After improving the SOCIO chatbot prototype model, we wanted to know how/if its usability has changed. An evidence-based empirical evaluation of the usability of SOCIO V1 (updated version) requires an extensive verification of the experimental results. A family of experiments is a method of verification whereby we can check if the experimental results are reproducible. Through comparison with the updated control tool Creately, we aimed to gain a better understanding of the usability of the collaborative modeling chatbot and how it could be improved based on experimental evidence of changes in terms of efficiency, effectiveness, satisfaction, and quality. A total of 87 students from three countries were recruited. We conducted a family of three experiments to compare the usability of SOCIO V1 and updated Creately in academic settings. Students appeared to be more satisfied with SOCIO V1, and SOCIO V1 scored better on completeness. There were no significant differences between the two tools regarding efficiency and quality. This study provides evidence on how to employ a family of experiments to improve chatbot usability and enrich knowledge on chatbot usability experimentation.


  • Open Access

    Asymob: a platform for measuring and clustering chatbots

    by Jose Maria Lopez-Morales, Pablo C. Canizares, Sara Perez-Soler, Esther Guerra, Juan de Lara

    Chatbots have become a popular way to access all sorts of services via natural language. Many platforms and tools have been proposed for their construction, like Google’s Dialogflow, Amazon’s Lex or Rasa. However, most of them still miss integrated quality assurance methods like metrics. Moreover, there is currently a lack of mechanisms to compare and classify chatbots possibly developed with heterogeneous technologies.To tackle these issues, we present Asymob, a web platform that enables the measurement of chatbots using a suite of 20 metrics. The tool features a repository supporting chatbots built with different technologies, like Dialogflow and Rasa. Asymob’s metrics help in detecting quality issues and serve to compare chatbots across and within technologies. The tool also helps in classifying chatbots along conversation topics or design features by means of two clustering methods: based on the chatbot metrics or on the phrases expected and produced by the chatbot. A video showcasing the tool is available at https://www.youtube.com/watch?v=8lpETkILpv8.


  • Choosing a Chatbot Development Tool

    by Sara Perez-Soler, Sandra Juarez-Puerta, Esther Guerra, Juan de Lara

    Chatbots are programs that supply services to users via conversation in natural language, acting as virtual assistants within social networks or web applications. Here, we review the most representative chatbot development tools with a focus on technical and managerial aspects.


  • Model-Driven Chatbot Development

    by Sara Pérez-Soler, Esther Guerra, Juan de Lara