A Bootstrap Pipeline for Chat-Based Image Retrieval with Effective Question Generation
Open Access
- 8 June 2026
- journal article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Multimedia Computing, Communications, and Applications
- Vol. 22 (6) , 1-24
Abstract
Chat-based image retrieval uses Large Language Models (LLM) to guide user input to enable more specific and precise search results, where LLM can enhance this process by asking user retrieval-oriented questions eliciting additional details about the target image. Despite the potential of this approach, no specialized Questioner model has been developed for this task due to the following significant challenges: (a) the difficulty of determining the optimal questions to ask; (b) the lack of a suitable protocol for fair model comparison; and (c) the notable scarcity of dialog-to-image retrieval data. To address these challenges, two fundamental principles are developed in this article to ensure the simplicity and effectiveness of the generated questions while enabling a fair comparison and accurate estimation of data quality and model performance. A bootstrap training methodology is introduced to collect retrieval-oriented dialog data and concurrently train the Questioner and the image Retriever. Under a fair comparison protocol, our extensive experiments have demonstrated that our proposed method can not only address the critical data gap but also achieve state-of-the-art results, which substantially surpass GPT-4o and GPT-4-Turbo through the fine-tuning of an 8B model.Keywords
This publication has 11 references indexed in Scilit:
- Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Models2024
- Evaluating the strengths and weaknesses of large language models in answering neurophysiology questionsScientific Reports, 2024
- ChatGPT for good? On opportunities and challenges of large language models for educationLearning and Individual Differences, 2023
- Cross-Modal Coherence for Text-to-Image RetrievalProceedings of the AAAI Conference on Artificial Intelligence, 2022
- COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval2022
- Visual Dialog2017
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- Microsoft COCO: Common Objects in Context2014
- Image Retrieval: Current Techniques, Promising Directions, and Open IssuesJournal of Visual Communication and Image Representation, 1999
- Relevance feedback: a power tool for interactive content-based image retrievalIEEE Transactions on Circuits and Systems for Video Technology, 1998