Multi-Modal and Multi-Domain Embedding Learning for Fashion Retrieval and Analysis
- 18 October 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 21 (6) , 1524-1537
Abstract
Big data analytics has been revolutionizing the fashion industry in recent years. This is evidenced by the fact that popular fashion brands and designers have relied on big data analytics to trace fashion trends and predict market patterns. In this work, we propose to learn a common latent feature representation from heterogeneous fashion data. Specifically, we design a multi-modal and multi-domain embedding learning framework for fashion analysis and data retrieval. Unlike most of the existing multi-view embedding methods, which only consider the heterogeneous similarity constraint, our proposed framework jointly considers both the homogeneous and heterogeneous similarity constraints to capture cross-view similarity and preserve the similarity of the same view. The proposed framework is comprised of two projection steps. In the first projection, a quintuplet-based ranking loss is proposed for multi-domain fashion data to preserve the homogeneous similarity. In the second projection, a cross-view similarity ranking loss is designed for multi-modal fashion data to capture heterogeneous similarity. By utilizing the learned common latent feature representation, the distance between any vector pairs from same or different modalities can reflect its semantic similarity. Quantitative evaluation on a new large-scale dataset and a fashion analysis case study demonstrate the effectiveness of our proposed method.Keywords
Funding Information
- National Basic Research Program of China (2015CB352400)
- National Science Foundation of China (61802100, 61836002, 61622205, 61472110, 61672455, 61528207, 61472348)
- Natural Science Foundation of Zhejiang Province of China (LY18F020005)
- National Research Foundation
- International Research Centre in Singapore Funding Initiative
This publication has 36 references indexed in Scilit:
- Image-Based Recommendations on Styles and Substitutes2015
- Learning visual similarity for product design with convolutional neural networksACM Transactions on Graphics, 2015
- Fashion Parsing With Video ContextIEEE Transactions on Multimedia, 2015
- Effective multi-modal retrieval based on stacked auto-encodersProceedings of the VLDB Endowment, 2014
- A Multi-View Embedding Space for Modeling Internet Images, Tags, and Their SemanticsInternational Journal of Computer Vision, 2013
- Apparel Classification with Style2013
- Multiview Metric Learning with Global Consistency and Local SmoothnessACM Transactions on Intelligent Systems and Technology, 2012
- Describing Clothing by Semantic Attributes2012
- Agglomerative Hierarchical Clustering with Constraints: Theoretical and Empirical Results2005
- Canonical Correlation Analysis: An Overview with Application to Learning MethodsNeural Computation, 2004