Weakly Supervised Coupled Networks for Visual Sentiment Analysis
Authors:
- Jufeng YangJYJufeng Yang
- Dongyu SheDSDongyu She
- Yu-Kun LaiYLYu-Kun Lai
- Paul L. RosinPRPaul L. Rosin
- Ming-Hsuan YangMYMing-Hsuan Yang
- 1 June 2018
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7584-7592
Abstract
Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions on-line. In this paper, we solve the problem of visual sentiment analysis using the high-level abstraction in the recognition process. Existing methods based on convolutional neural networks learn sentiment representations from the holistic image appearance. However, different image regions can have a different influence on the intended expression. This paper presents a weakly supervised coupled convolutional network with two branches to leverage the localized information. The first branch detects a sentiment specific soft map by training a fully convolutional network with the cross spatial pooling strategy, which only requires image-level labels, thereby significantly reducing the annotation burden. The second branch utilizes both the holistic and localized information by coupling the sentiment map with deep features for robust classification. We integrate the sentiment detection and classification branches into a unified deep framework and optimize the network in an end-to-end manner. Extensive experiments on six benchmark datasets demonstrate that the proposed method performs favorably against the state-of-the-art methods for visual sentiment analysis.Keywords
This publication has 32 references indexed in Scilit:
- Predicting Personalized Image Emotion Perceptions in Social NetworksIEEE Transactions on Affective Computing, 2016
- Where do emotions come from? Predicting the Emotion Stimuli Map2016
- Fast R-CNN2015
- Is object localization for free? - Weakly-supervised learning with convolutional neural networks2015
- A mixed bag of emotions: Model, predict, and transfer emotion distributions2015
- Exploring Principles-of-Art Features For Image Emotion Recognition2014
- Large-scale visual sentiment ontology and detectors using adjective noun pairs2013
- Context-aware affective images classification based on bilayer sparse representation2012
- Measuring the Objectness of Image WindowsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
- LIBSVMACM Transactions on Intelligent Systems and Technology, 2011