FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 14085-14095
Abstract
Fashion image retrieval based on a query pair of reference image and natural language feedback is a challenging task that requires models to assess fashion related information from visual and textual modalities simultaneously. We propose a new vision-language transformer based model, FashionVLP, that brings the prior knowledge contained in large image-text corpora to the domain of fashion image retrieval, and combines visual information from multiple levels of context to effectively capture fashion-related information. While queries are encoded through the transformer layers, our asymmetric design adopts a novel attention-based approach for fusing target image features without involving text or transformer layers in the process. Extensive results show that FashionVLP achieves the state-of-the-art performance on benchmark datasets, with a large 23% relative improvement on the challenging FashionIQ dataset, which contains complex natural language feedback.Keywords
This publication has 38 references indexed in Scilit:
- CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples2016
- Deep Image Retrieval: Learning Global Representations for Image Search2016
- Fashion Landmark Detection in the Wild2016
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep Residual Learning for Image Recognition2016
- NetVLAD: CNN Architecture for Weakly Supervised Place Recognition2016
- FaceNet: A unified embedding for face recognition and clustering2015
- Automatic Attribute Discovery and Characterization from Noisy Web Data2010
- Learning a Similarity Metric Discriminatively, with Application to Face Verification2005
- Eigenfaces for RecognitionJournal of Cognitive Neuroscience, 1991