Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning
Open Access
- 26 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 3337-3345
Abstract
Multimodal movie genre classification has always been regarded as a demanding multi-label classification task due to the diversity of multimodal data such as posters, plot summaries, trailers and metadata. Although existing works have made great progress in modeling and combining each modality, they still face three issues: 1) unutilized group relations in metadata, 2) unreliable attention allocation, and 3) indiscriminative fused features. Given that the knowledge graph has been proven to contain rich information, we present a novel framework that exploits the knowledge graph from various perspectives to address the above problems. As a preparation, the metadata is processed into a domain knowledge graph. A translate model for knowledge graph embedding is adopted to capture the relations between entities. Firstly we retrieve the relevant embedding from the knowledge graph by utilizing group relations in metadata and then integrate it with other modalities. Next, we introduce an Attention Teacher module for reliable attention allocation based on self-supervised learning. It learns the distribution of the knowledge graph and produces rational attention weights. Finally, a Genre-Centroid Anchored Contrastive Learning module is proposed to strengthen the discriminative ability of fused features. The embedding space of anchors is initialized from the genre entities in the knowledge graph. To verify the effectiveness of our framework, we collect a larger and more challenging dataset named MM-IMDb 2.0 compared with the MM-IMDb dataset. The experimental results on two datasets demonstrate that our model is superior to the state-of-the-art methods. Our code and dataset is available at https://github.com/aoluming/IDKG.git.Keywords
All Related Versions
This publication has 22 references indexed in Scilit:
- Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models2020
- CentralNet: A Multilayer Approach for Multimodal Fusion2019
- Tracking Emerges by Colorizing Videos2018
- Deep Multimodal Features for Movie Genre and Interestingness Prediction2018
- OpenKE: An Open Toolkit for Knowledge Embedding2018
- Movie Genre Classification from Plot Summaries Using Bidirectional LSTM2018
- Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification2016
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Movie genre classification with Convolutional Neural Networks2016
- A movie recommendation algorithm based on genre correlationsExpert Systems with Applications, 2012