Cross-Modal Variational Auto-Encoder for Content-Based Micro-Video Background Music Recommendation
- 16 November 2021
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 515-528
Abstract
In this paper, we propose a cross-modal variational auto-encoder (CMVAE) for content-based micro-video background music recommendation. CMVAE is a hierarchical Bayesian generative model that matches relevant background music to a micro-video by projecting these two multimodal inputs into a shared low-dimensional latent space, where the alignment of two corresponding embeddings of a matched video-music pair is achieved by cross-generation. Moreover, the multimodal information is fused by the product-of-experts (PoE) principle, where the semantic information in visual and textual modalities of the micro-video are weighted according to their variance estimations such that the modality with a lower noise level is given more weights. Therefore, the micro-video latent variables contain less irrelevant information that results in a more robust model generalization. Furthermore, we establish a large-scale content-based micro-video background music recommendation dataset, TT-150k, composed of extracted features from approximately 3,000 different background music clips associated to 150,000 micro-videos from different users. Extensive experiments on the established TT-150k dataset demonstrate the effectiveness of the proposed method. A qualitative assessment of CMVAE by visualizing some recommendation results is also included.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62036005)
This publication has 63 references indexed in Scilit:
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models2015
- Affective Music Recommendation System Based on the Mood of Input VideoLecture Notes in Computer Science, 2015
- ADVISOR2014
- Semantic Based Background Music Recommendation for Home Videos2014
- Cross matching of music and image2012
- Introduction to the special issue on Theories and Methodologies for mechatronics designMechatronics, 2010
- NUS-WIDE2009
- The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathologyDevelopment and Psychopathology, 2005
- Canonical Correlation Analysis: An Overview with Application to Learning MethodsNeural Computation, 2004
- Information Theory and Statistical Mechanics. IIPhysical Review B, 1957