Explainable Outfit Recommendation with Joint Outfit Matching and Comment Generation
- 19 March 2019
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 32 (8) , 1502-1516
Abstract
Most previous work on outfit recommendation focuses on designing visual features to enhance recommendations. Existing work neglects user comments of fashion items, which have been proved to be effective in generating explanations along with better recommendation results. We propose a novel neural network framework, neural outfit recommendation (NOR), that simultaneously provides outfit recommendations and generates abstractive comments. NOR consists of two parts: outfit matching and comment generation. For outfit matching, we propose a convolutional neural network with a mutual attention mechanism to extract visual features. The visual features are then decoded into a rating score for the matching prediction. For abstractive comment generation, we propose a gated recurrent neural network with a cross-modality attention mechanism to transform visual features into a concise sentence. The two parts are jointly trained based on a multi-task learning framework in an end-to-end back-propagation paradigm. Extensive experiments conducted on an existing dataset and a collected real-world dataset show NOR achieves significant improvements over state-of-the-art baselines for outfit recommendation. Meanwhile, our generated comments achieve impressive ROUGE and BLEU scores in comparison to human-written comments. The generated comments can be regarded as explanations for the recommendation results. We release the dataset and code to facilitate future research.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (61672324, 61672322)
- Natural Science Foundation of Shandong Province (2016ZRE27468)
- Fundamental Research Funds of Shandong University
- Ahold Delhaize
- Association of Universities in the Netherlands
- Innovation Center for Artificial Intelligence
This publication has 50 references indexed in Scilit:
- Image-Based Recommendations on Styles and Substitutes2015
- Retrieving Similar Styles to Parse ClothingIEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
- On the Properties of Neural Machine Translation: Encoder–Decoder Approaches2014
- Personalized Recommendation Combining User Interest and Social CircleIEEE Transactions on Knowledge and Data Engineering, 2013
- Hi, magic closet, tell me what to wear!2012
- Mapping queries to the Linking Open Data cloud: A case study using DBpediaJournal of Web Semantics, 2011
- An effective coherence measure to determine topical consistency in user-generated contentInternational Journal on Document Analysis and Recognition (IJDAR), 2009
- Pharaoh: A Beam Search Decoder for Phrase-Based Statistical Machine Translation Models2004
- Multitask Learning1998
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998