Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models
Open Access
- 30 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 978-987
Abstract
Image search stands as a pivotal task in multimedia and computer vision, finding applications across diverse domains, ranging from internet search to medical diagnostics. Conventional image search systems operate by accepting textual or visual queries, retrieving the top-relevant candidate results from the database. However, prevalent methods often rely on single-turn procedures, introducing potential inaccuracies and limited recall. These methods also face the challenges, such as vocabulary mismatch and the semantic gap, constraining their overall effectiveness. To address these issues, we propose an interactive image retrieval system capable of refining queries based on user relevance feedback in a multi-turn setting. This system incorporates a vision language model (VLM) based image captioner to enhance the quality of text-based queries, resulting in more informative queries with each iteration. Moreover, we introduce a large language model (LLM) based denoiser to refine text-based query expansions, mitigating inaccuracies in image descriptions generated by captioning models. To evaluate our system, we curate a new dataset by adapting the MSR-VTT video retrieval dataset to the image retrieval task, offering multiple relevant ground truth images for each query. Through comprehensive experiments, we validate the effectiveness of our proposed system against baseline methods, achieving state-of-the-art performance with a notable 10% improvement in terms of recall. Our contributions encompass the development of an innovative interactive image retrieval system, the integration of an LLM-based denoiser, the curation of a meticulously designed evaluation dataset, and thorough experimental validation.Keywords
All Related Versions
This publication has 59 references indexed in Scilit:
- Interactive Learning for Multimedia at Large2020
- Aggregation of Multiple Pseudo Relevance Feedbacks for Image Search Re-RankingIEEE Access, 2019
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence ModelsInternational Journal of Computer Vision, 2016
- Learning a Deep Convolutional Network for Image Super-Resolution2014
- Leveraging visual concepts and query performance prediction for semantic-theme-based video retrievalInternational Journal of Multimedia Information Retrieval, 2012
- Recommender Systems Handbook2011
- The retrieval effectiveness of web search engines: considering results descriptionsJournal of Documentation, 2008
- IM2GPS: estimating geographic information from a single image2008
- Content-based image retrieval at the end of the early yearsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- ON TERM SELECTION FOR QUERY EXPANSIONJournal of Documentation, 1990