CoVR: Learning Composed Video Retrieval from Web Video Captions
Authors:
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (6) , 5270-5279
Abstract
Composed Image Retrieval (CoIR) has recently gained popularity as a task that considers both text and image queries together, to search for relevant images in a database. Most CoIR approaches require manually annotated datasets, comprising image-text-image triplets, where the text describes a modification from the query image to the target image. However, manual curation of CoIR triplets is expensive and prevents scalability. In this work, we instead propose a scalable automatic dataset creation methodology that generates triplets given video-caption pairs, while also expanding the scope of the task to include composed video retrieval (CoVR). To this end, we mine paired videos with a similar caption from a large database, and leverage a large language model to generate the corresponding modification text. Applying this methodology to the extensive WebVid2M collection, we automatically construct our WebVid-CoVR dataset, resulting in 1.6 million triplets. Moreover, we introduce a new benchmark for CoVR with a manually annotated evaluation set, along with baseline results. Our experiments further demonstrate that training a CoVR model on our dataset effectively transfers to CoIR, leading to improved state-of-the-art performance in the zero-shot setup on both the CIRR and FashionIQ benchmarks. Our code, datasets, and models are publicly available at https://imagine.enpc.fr/~ventural/covr.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Align and Prompt: Video-and-Language Pre-training with Entity Prompts2022
- Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions2022
- Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback2021
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- UNITER: UNiversal Image-TExt Representation Learning2020
- End-to-End Learning of Visual Representations From Uncurated Instructional Videos2020
- Composing Text and Image for Image Retrieval - an Empirical Odyssey2019
- Microsoft COCO: Common Objects in Context2014