Probing Sentiment-Oriented PreTraining Inspired by Human Sentiment Perception Mechanism
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 2850-2860
Abstract
Pre-training of deep convolutional neural networks (DC-NNs) plays a crucial role in the field of visual sentiment analysis (VSA). Most proposed methods employ the off-the-shelf backbones pre-trained on large-scale object classification datasets (i.e., ImageNet). While it boosts performance for a big margin against initializing model states from random, we argue that DCNNs simply pre-trained on ImageNet may excessively focus on recognizing objects, but failed to provide high-level concepts in terms of sentiment. To address this long-term overlooked problem, we propose a sentiment-oriented pre-training method that is built upon human visual sentiment perception (VSP) mechanism. Specifically, we factorize the process of VSP into three steps, namely stimuli taking, holistic organizing, and high-level perceiving. From imitating each VSP step, a total of three models are separately pre-trained via our devised sentiment-aware tasks that contribute to excavating sentiment-discriminated representations. Moreover, along with our elaborated multi-model amalgamation strategy, the prior knowledge learned from each perception step can be effectively transferred into a single target model, yielding substantial performance gains. Finally, we verify the superiorities of our proposed method over extensive experiments, covering mainstream VSA tasks from single-label learning (SLL), multi-label learning (MLL), to label distribution learning (LDL). Experiment results demonstrate that our proposed method leads to unanimous improvements in these downstream tasks. Our code is released on https://github.com/tinglyfeng/sentiment_pretraining.Keywords
Funding Information
- National Key Research and Development Program of China (2018AAA0100400)
- Fundamental Research Funds for the Central Universities
This publication has 41 references indexed in Scilit:
- A mixed bag of emotions: Model, predict, and transfer emotion distributions2015
- Object-Based Visual Sentiment Concept Analysis and Application2014
- The Neuronal Organization of the RetinaNeuron, 2012
- The brain basis of emotion: A meta-analytic reviewBehavioral and Brain Sciences, 2012
- Spatially Global Representations in Human Primary Visual Cortex during Working Memory MaintenanceThe Journal of Neuroscience, 2009
- Image emotional classification: static vs. dynamic2005
- THE HUMAN VISUAL CORTEXAnnual Review of Neuroscience, 2004
- Vision, emotion and memory: from neurophysiology to computationInternational Congress Series, 2003
- An argument for basic emotionsCognition and Emotion, 1992
- The role of emotion in advertisingPsychology & Marketing, 1984