MDA: Multimodal Data Augmentation Framework for Boosting Performance on Sentiment/Emotion Classification Tasks
- 25 September 2020
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Intelligent Systems
- Vol. 36 (6) , 3-12
Abstract
Multimodal data analysis has drawn increasing attention with the explosive growth of multimedia data. Although traditional unimodal data analysis tasks have accumulated abundant labeled datasets, there are few labeled multimodal datasets due to the difficulty and complexity of multimodal data annotation, nor is it easy to directly transfer unimodal knowledge into multimodal data. Unfortunately, there are only a little related data augmentation work in multimodal domain, especially for image-text data. In this paper, to address the scarcity problem of labeled multimodal data, we propose a Multimodal Data Augmentation (MDA) framework for boosting performance on multimodal classification task. Our framework learns a cross-modality matching network to select image-text pairs from existing unimodal datasets as the multimodal synthetic dataset, and uses this dataset to enhance the performance of classifiers. We take the multimodal sentiment analysis and multimodal emotion analysis as the experimental tasks and the experimental results show the effectiveness of our framework for boosting the performance on multimodal classification task.Keywords
Funding Information
- Ministry of Science and Technology of China (2016QY02D0305, 2018ZX10201001)
- National Natural Science Foundation of China (71621002, 11832001)
This publication has 16 references indexed in Scilit:
- Texygen2018
- A Co-Memory Network for Multimodal Sentiment Analysis2018
- Ensemble learning on visual and textual data for social image emotion classificationInternational Journal of Machine Learning and Cybernetics, 2017
- Unlabeled Samples Generated by GAN Improve the Person Re-identification Baseline in Vitro2017
- Person Search with Natural Language Description2017
- The Impact of Content, Context, and Creator on User Engagement in Social Media Marketing2017
- Tensor Fusion Network for Multimodal Sentiment Analysis2017
- Sentiment Analysis on Multi-View Social Data2016
- Large-scale visual sentiment ontology and detectors using adjective noun pairs2013
- Boosting for transfer learning2007