GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
Open Access
- 24 August 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4951-4961
Abstract
We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.Keywords
All Related Versions
This publication has 16 references indexed in Scilit:
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Perceived conversational ability of task-based chatbots – Which conversational elements influence the success of text-based dialogues?International Journal of Information Management, 2023
- Exploiting Simulated User Feedback for Conversational Search: Ranking, Rewriting, and Beyond2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- Conversational Information Seeking2023
- VILT2022
- Conversational question answering: a surveyKnowledge and Information Systems, 2022
- A Word Embedding-Based Method for Unsupervised Adaptation of Cooking RecipesIEEE Access, 2021
- UNIFIEDQA: Crossing Format Boundaries with a Single QA System2020
- Genie: a generator of natural language semantic parsers for virtual assistant commands2019