EmoSEM: Segment and Explain Emotion Stimuli in Visual Art
Preprint
Abstract
This paper focuses on a key challenge in visual art understanding: given an art image, the model pinpoints pixel regions that trigger a specific human emotion, and generates linguistic explanations for the emotional arousal. Despite recent advances in art understanding, pixel-level emotion understanding still faces a dual challenge: first, the subjectivity of emotion makes it difficult for general segmentation models like SAM to adapt to emotion-oriented segmentation tasks; and second, the abstract nature of art expression makes it difficult for captioning models to balance pixel-level semantic understanding and emotion reasoning. To solve the above problems, this paper proposes the Emotion stimuli Segmentation and Explanation Model (EmoSEM) to endow the segmentation model SAM with emotion comprehension capability. First, to enable the model to perform segmentation under the guidance of emotional intent well, we introduce an emotional prompt with a learnable mask token as the conditional input for segmentation decoding. Then, we design an emotion projector to establish the association between emotion and visual features. Next, more importantly, to address emotion-visual stimuli alignment, we develop a lightweight prefix projector, a module that fuses the learned emotional mask with the corresponding emotion into a unified representation compatible with the language model.Finally, we input the joint visual, mask, and emotional tokens into the language model and output the emotional explanations. It ensures that the generated interpretations remain semantically and emotionally coherent with the visual stimuli. The method innovatively realizes end-to-end modeling from low-level pixel features to high-level emotion interpretation, providing the first interpretable fine-grained analysis framework for artistic emotion computing. Extensive experiments validate the effectiveness of our model.Keywords
All Related Versions
This publication has 19 references indexed in Scilit:
- MASANet: Multi-Aspect Semantic Auxiliary Network for Visual Sentiment AnalysisIEEE Transactions on Affective Computing, 2024
- Sentimental Visual Captioning using Multimodal TransformerInternational Journal of Computer Vision, 2023
- Simple But Powerful, a Language-Supervised Method for Image Emotion ClassificationIEEE Transactions on Affective Computing, 2022
- Scaling Open-Vocabulary Image Segmentation with Image-Level Labels2022
- It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection2022
- Image Captioning with Inherent Sentiment2021
- ArtEmis: Affective Language for Visual Art2021
- Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers2021
- EmoSen: Generating Sentiment and Emotion Controlled Responses in a Multimodal Dialogue SystemIEEE Transactions on Affective Computing, 2020
- Deep Residual Learning for Image Recognition2016