Comparative Analysis of Generic and Fine-Tuned Large Language Models for Conversational Agent Systems
Open Access
Abstract
In the rapidly evolving domain of conversational agents, the integration of Large Language Models (LLMs) into Chatbot Development Platforms (CDPs) is a significant innovation. This study compares the efficacy of employing generic and fine-tuned GPT-3.5-turbo models for designing dialog flows, focusing on the intent and entity recognition crucial for dynamic conversational interactions. Two distinct approaches are introduced: a generic GPT-based system (G-GPT) leveraging the pre-trained model with complex prompts for intent and entity detection, and a fine-tuned GPT-based system (FT-GPT) employing customized models for enhanced specificity and efficiency. The evaluation encompassed the systems’ ability to accurately classify intents and recognize named entities, contrasting their adaptability, operational efficiency, and customization capabilities. The results revealed that, while the G-GPT system offers ease of deployment and versatility across various contexts, the FT-GPT system demonstrates superior precision, efficiency, and customization, although it requires initial training and dataset preparation. This research highlights the versatility of LLMs in enriching conversational features for talking assistants, from social robots to interactive chatbots. By tailoring these advanced models, the fluidity and responsiveness of conversational agents can be enhanced, making them more adaptable and effective in a variety of settings, from customer service to interactive learning environments.Keywords
Funding Information
- Junta de Comunidades de Castilla-La Mancha (SBPLY/21/180501/000160, TED2021-130296A-I00, PDC2022-133457-I00, FPU22/00839, 2022-PRED-20651)
- Agencia Estatal de Investigación (SBPLY/21/180501/000160, TED2021-130296A-I00, PDC2022-133457-I00, FPU22/00839, 2022-PRED-20651)
- Ministerio de Ciencia e Innovación (SBPLY/21/180501/000160, TED2021-130296A-I00, PDC2022-133457-I00, FPU22/00839, 2022-PRED-20651)
- Universidad de Castilla-La Mancha (SBPLY/21/180501/000160, TED2021-130296A-I00, PDC2022-133457-I00, FPU22/00839, 2022-PRED-20651)
This publication has 19 references indexed in Scilit:
- CPMI-ChatGLM: parameter-efficient fine-tuning ChatGLM with Chinese patent medicine instructionsScientific Reports, 2024
- Ecosystem for Construction of Hybrid Conversational Systems (BRasa)2024
- ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scopeInternet of Things and Cyber-Physical Systems, 2023
- HCI International 2022 – Late Breaking Posters2022
- PACAs: Process-Aware Conversational Agents2022
- Florence- A Health Care Chatbot2021
- Development of a Voice Chatbot for Payment Using Amazon Lex Service with Eyowo as the Payment Platform2020
- Xatkit: A Multimodal Low-Code Chatbot Development FrameworkIEEE Access, 2020
- A Voice Interactive Multilingual Student Support System using IBM Watson2019
- Conversational Agent in mHealth to empower people managing the Parkinson’s DiseaseProcedia Computer Science, 2019