Carlos Gemmell
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Carlos Gemmell has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
- Jeffrey Dalton
The University of Edinburgh, Edinburgh, United Kingdom
4 shared publications - Iain Mackie
University of Glasgow University of Glasgow University of Glasgow University of Glasgow
2 shared publications - Paul Owoicho
University of Glasgow Glasgow Scotland UK
2 shared publications - Sophie Fischer
University of Glasgow Glasgow Scotland UK
2 shared publications - Federico Rossetto
University of Glasgow, Glasgow, United Kingdom
1 shared publication - Federico Rossetto
University of Glasgow Glasgow Scotland UK
1 shared publication
Latest Publications
- Open Access
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 4951-4961
GRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 4951-4961
Open AccessGRILLBot In Practice: Lessons and Tradeoffs Deploying Large Language Models for Adaptable Conversational Task Assistants
We tackle the challenge of building real-world multimodal assistants for complex real-world tasks. We describe the practicalities and challenges of developing and deploying GRILLBot, a leading (first and second prize winning in 2022 and 2023) system deployed in the Alexa Prize TaskBot Challenge. Building on our Open Assistant Toolkit (OAT) framework, we propose a hybrid architecture that leverages Large Language Models (LLMs) and specialised models tuned for specific subtasks requiring very low latency. OAT allows us to define when, how and which LLMs should be used in a structured and deployable manner. For knowledge-grounded question answering and live task adaptations, we show that LLM reasoning abilities over task context and world knowledge outweigh latency concerns. For dialogue state management, we implement a code generation approach and show that specialised smaller models have 84% effectiveness with 100x lower latency. Overall, we provide insights and discuss tradeoffs for deploying both traditional models and LLMs to users in complex real-world multimodal environments in the Alexa TaskBot challenge. These experiences will continue to evolve as LLMs become more capable and efficient -- fundamentally reshaping OAT and future assistant architectures.
- Open Access
Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Data
Tabular question answering (TQA) presents a challenging setting for neural systems by requiring joint reasoning of natural language with large amounts of semi-structured data. Unlike humans who use programmatic tools like filters to transform data before processing, language models in TQA process tables directly, resulting in information loss as table size increases. In this paper we propose ToolWriter to generate query specific programs and detect when to apply them to transform tables and align them with the TQA model's capabilities. Focusing ToolWriter to generate row-filtering tools improves the state-of-the-art for WikiTableQuestions and WikiSQL with the most performance gained on long tables. By investigating headroom, our work highlights the broader potential for programmatic tools combined with neural components to manipulate large amounts of structured data.
Open AccessGenerate, Transform, Answer: Question Specific Tool Synthesis for Tabular Data
Tabular question answering (TQA) presents a challenging setting for neural systems by requiring joint reasoning of natural language with large amounts of semi-structured data. Unlike humans who use programmatic tools like filters to transform data before processing, language models in TQA process tables directly, resulting in information loss as table size increases. In this paper we propose ToolWriter to generate query specific programs and detect when to apply them to transform tables and align them with the TQA model's capabilities. Focusing ToolWriter to generate row-filtering tools improves the state-of-the-art for WikiTableQuestions and WikiSQL with the most performance gained on long tables. By investigating headroom, our work highlights the broader potential for programmatic tools combined with neural components to manipulate large amounts of structured data.
- Open Access
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 16137-16148
ToolWriter: Question Specific Tool Synthesis for Tabular Data
Carlos Gemmell, Jeff Dalton. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- 1 January 2023
- Published by Association for Computational Linguistics (ACL)
- p. 16137-16148
Open AccessToolWriter: Question Specific Tool Synthesis for Tabular Data
Carlos Gemmell, Jeff Dalton. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.
- Open Access
CODEC: Complex Document and Entity Collection
CODEC is a document and entity ranking benchmark that focuses on complex research topics. We target essay-style information needs of social science researchers, i.e. "How has the UK's Open Banking Regulation benefited Challenger Banks?". CODEC includes 42 topics developed by researchers and a new focused web corpus with semantic annotations including entity links. This resource includes expert judgments on 17,509 documents and entities (416.9 per topic) from diverse automatic and interactive manual runs. The manual runs include 387 query reformulations, providing data for query performance prediction and automatic rewriting evaluation. CODEC includes analysis of state-of-the-art systems, including dense retrieval and neural re-ranking. The results show the topics are challenging with headroom for document and entity ranking improvement. Query expansion with entity information shows significant gains in document ranking, demonstrating the resource's value for evaluating and improving entity-oriented search. We also show that the manual query reformulations significantly improve document ranking and entity ranking performance. Overall, CODEC provides challenging research topics to support the development and evaluation of entity-centric search methods.
Open AccessCODEC: Complex Document and Entity Collection
CODEC is a document and entity ranking benchmark that focuses on complex research topics. We target essay-style information needs of social science researchers, i.e. "How has the UK's Open Banking Regulation benefited Challenger Banks?". CODEC includes 42 topics developed by researchers and a new focused web corpus with semantic annotations including entity links. This resource includes expert judgments on 17,509 documents and entities (416.9 per topic) from diverse automatic and interactive manual runs. The manual runs include 387 query reformulations, providing data for query performance prediction and automatic rewriting evaluation. CODEC includes analysis of state-of-the-art systems, including dense retrieval and neural re-ranking. The results show the topics are challenging with headroom for document and entity ranking improvement. Query expansion with entity information shows significant gains in document ranking, demonstrating the resource's value for evaluating and improving entity-oriented search. We also show that the manual query reformulations significantly improve document ranking and entity ranking performance. Overall, CODEC provides challenging research topics to support the development and evaluation of entity-centric search methods.
- Open Access
- 1 January 2022
- Published by Association for Computational Linguistics (ACL)
- p. 654-658
GRILLBot: A multi-modal conversational agent for complex real-world tasks
Carlos Gemmell, Federico Rossetto, Iain Mackie, Paul Owoicho, Sophie Fischer, Jeff Dalton. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.
- 1 January 2022
- Published by Association for Computational Linguistics (ACL)
- p. 654-658
Open AccessGRILLBot: A multi-modal conversational agent for complex real-world tasks
Carlos Gemmell, Federico Rossetto, Iain Mackie, Paul Owoicho, Sophie Fischer, Jeff Dalton. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue. 2022.