Data Roaming and Quality Assessment for Composed Image Retrieval
Authors:
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (4) , 2991-2999
Abstract
The task of Composed Image Retrieval (CoIR) involves queries that combine image and text modalities, allowing users to express their intent more effectively. However, current CoIR datasets are orders of magnitude smaller compared to other vision and language (V&L) datasets. Additionally, some of these datasets have noticeable issues, such as queries containing redundant modalities. To address these shortcomings, we introduce the Large Scale Composed Image Retrieval (LaSCo) dataset, a new CoIR dataset which is ten times larger than existing ones. Pre-training on our LaSCo, shows a noteworthy improvement in performance, even in zero-shot. Furthermore, we propose a new approach for analyzing CoIR datasets and methods, which detects modality redundancy or necessity, in queries. We also introduce a new CoIR baseline, the Cross-Attention driven Shift Encoder (CASE). This baseline allows for early fusion of modalities using a cross-attention module and employs an additional auxiliary task during training. Our experiments demonstrate that this new baseline outperforms the current state-of-the-art methods on established benchmarks like FashionIQ and CIRR.Keywords
All Related Versions
This publication has 23 references indexed in Scilit:
- Classification-Regression for Chart Comprehension2022
- Learning Audio-Video Modalities from Image Captions2022
- Content-Based Image Retrieval and the Semantic Gap in the Deep Learning Era2021
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- Composed Query Image Retrieval Using Locally Bounded Features2020
- Image Search With Text Feedback by Visiolinguistic Attention Learning2020
- Composing Text and Image for Image Retrieval - an Empirical Odyssey2019
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Discovering states and transformations in image collections2015
- Microsoft COCO: Common Objects in Context2014