GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
Authors:
- Sophie FischerSFSophie Fischer
- Carlos GemmellCGCarlos Gemmell
- Niklas TecklenburgNTNiklas Tecklenburg
- Iain MackieIMIain Mackie
- Federico RossettoFRFederico Rossetto
- Jeffrey DaltonJDJeffrey Dalton
Preprint
Abstract
We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.Keywords
All Related Versions
This publication has 6 references indexed in Scilit:
- A survey on large language model based autonomous agentsFrontiers of Computer Science, 2024
- Perceived conversational ability of task-based chatbots – Which conversational elements influence the success of text-based dialogues?International Journal of Information Management, 2023
- Survey of Hallucination in Natural Language GenerationACM Computing Surveys, 2023
- A Word Embedding-Based Method for Unsupervised Adaptation of Cooking RecipesIEEE Access, 2021
- Exploiting Food Embeddings for Ingredient Substitution2021
- Recipe1M+: A Dataset for Learning Cross-Modal Embeddings for Cooking Recipes and Food ImagesIEEE Transactions on Pattern Analysis and Machine Intelligence, 2019