Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning
Authors:
- Jiaqi LiJLJiaqi Li
- Guilin QiGQGuilin Qi
- Chuanyi ZhangCZChuanyi Zhang
- Yongrui ChenYCYongrui Chen
- Yiming TanYTYiming Tan
- Chenlong XiaCXChenlong Xia
- Ye TianYTYe Tian
Preprint
Abstract
Multimodal movie genre classification has always been regarded as a demanding multi-label classification task due to the diversity of multimodal data such as posters, plot summaries, trailers and metadata. Although existing works have made great progress in modeling and combining each modality, they still face three issues: 1) unutilized group relations in metadata, 2) unreliable attention allocation, and 3) indiscriminative fused features. Given that the knowledge graph has been proven to contain rich information, we present a novel framework that exploits the knowledge graph from various perspectives to address the above problems. As a preparation, the metadata is processed into a domain knowledge graph. A translate model for knowledge graph embedding is adopted to capture the relations between entities. Firstly we retrieve the relevant embedding from the knowledge graph by utilizing group relations in metadata and then integrate it with other modalities. Next, we introduce an Attention Teacher module for reliable attention allocation based on self-supervised learning. It learns the distribution of the knowledge graph and produces rational attention weights. Finally, a Genre-Centroid Anchored Contrastive Learning module is proposed to strengthen the discriminative ability of fused features. The embedding space of anchors is initialized from the genre entities in the knowledge graph. To verify the effectiveness of our framework, we collect a larger and more challenging dataset named MM-IMDb 2.0 compared with the MM-IMDb dataset. The experimental results on two datasets demonstrate that our model is superior to the state-of-the-art methods. We will release the code in the near future.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- Multimodal movie genre classification using recurrent neural networkMultimedia Tools and Applications, 2022
- Learning discriminative representations for multi-label image recognitionJournal of Visual Communication and Image Representation, 2022
- Image-Text Integration Using a Multimodal Fusion Network Module for Movie Genre ClassificationIET Conference Proceedings, 2021
- Condensed Movies: Story Based Retrieval with Contextual Embeddings2021
- MovieNet: A Holistic Dataset for Movie Understanding2020
- Self-Supervised Learning of Pretext-Invariant Representations2020
- Tracking Emerges by Colorizing Videos2018
- Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification2016
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- A movie recommendation algorithm based on genre correlationsExpert Systems with Applications, 2012