Generalizable Tip-of-the-Tongue Retrieval with LLM Re-ranking
Open Access
- 10 July 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 2437-2441
Abstract
Tip-of-the-Tongue (ToT) retrieval is challenging for search engines because the queries are usually natural-language, verbose, and contain uncertain and inaccurate information. This paper studies the generalization capabilities of existing retrieval methods with ToT queries in multiple domains. We curate a multi-domain dataset and evaluate the effectiveness of recall-oriented first-stage retrieval methods across the different domains, considering in-domain, out-of-domain, and multi-domain training settings. We further explore the use of a Large Language Model (LLM), i.e. GPT-4, for zero-shot re-ranking in various ToT domains, relying solely on the item titles. Results show that multi-domain training enhances recall, and that LLMs are strong zero-shot re-rankers, especially for popular items, outperforming direct GPT-4 prompting without first-stage retrieval. Datasets and code can be found on GitHub https://github.com/LuisPB7/TipTongueKeywords
Funding Information
- Fundação para a Ciência e Tecnologia (FCT) (SFRH/BD/150497/2019)
- PIDDAC programme (UIDB/50021/2020)
- Portuguese Recovery and Resilience Plan (C645008882-00000055)
This publication has 11 references indexed in Scilit:
- KALE: Using a K-Sparse Projector for Lexical Expansion2023
- A Unified Framework for Learned Sparse Retrieval2023
- 'It's on the tip of my tongue'2022
- Large Dual Encoders Are Generalizable Retrievers2022
- Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrieval2022
- Towards Understanding Complex Known-Item Requests on Reddit2021
- Comparing Score Aggregation Approaches for Document Retrieval with Pretrained Transformers2021
- Tip of the Tongue Known-Item Retrieval2021
- Dense Passage Retrieval for Open-Domain Question Answering2020
- Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval1994