MetaCLUE: Towards Comprehensive Visual Metaphors Research
Authors:
- Arjun R. AkulaAAArjun R. Akula
- Brendan DriscollBDBrendan Driscoll
- Pradyumna NarayanaPNPradyumna Narayana
- Soravit ChangpinyoSCSoravit Changpinyo
- Zhiwei JiaZJZhiwei Jia
- Suyash DamleSDSuyash Damle
- Garima PruthiGPGarima Pruthi
- Sugato BasuSBSugato Basu
- Leonidas GuibasLGLeonidas Guibas
- William T. FreemanWFWilliam T. Freeman
- Yuanzhen LiYLYuanzhen Li
- Varun JampaniVJVarun Jampani
Preprint
Abstract
Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- A Survey on Computational Metaphor ProcessingACM Computing Surveys, 2020
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question AnsweringInternational Journal of Computer Vision, 2018
- SPICE: Semantic Propositional Image Caption Evaluation2016
- An Empirical Study on the Role of Perceptual Similarity in Visual Metaphors and CreativityMetaphor and Symbol, 2013
- Towards supporting evocation processes in creative design: A cognitive approachInternational Journal of Human-Computer Studies, 2005
- CorMet: A Computational, Corpus-Based Conventional Metaphor Extraction SystemComputational Linguistics, 2004
- The Impact of Verbal Anchoring on Consumer Response to Image AdsJournal of Advertising, 2000
- Images in Advertising: The Need for a Theory of Visual RhetoricJournal of Consumer Research, 1994