MetaCLUE: Towards Comprehensive Visual Metaphors Research
Authors:
- 1 June 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 23201-23211
Abstract
Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce Meta-CLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor classification, localization, understanding (retrieval, question answering, captioning) and generation (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities. Project page: https://metaclue.github.ioKeywords
All Related Versions
This publication has 32 references indexed in Scilit:
- An Empirical Study on the Role of Perceptual Similarity in Visual Metaphors and CreativityMetaphor and Symbol, 2013
- A corpus-based analysis of context effects on metaphor comprehension2006
- Towards supporting evocation processes in creative design: A cognitive approachInternational Journal of Human-Computer Studies, 2005
- Beyond Visual Metaphor: A New Typology of Visual Rhetoric in AdvertisingMarketing Theory, 2004
- CorMet: A Computational, Corpus-Based Conventional Metaphor Extraction SystemComputational Linguistics, 2004
- Similarity and Emergence in Conceptual CombinationJournal of Memory and Language, 2001
- The Impact of Verbal Anchoring on Consumer Response to Image AdsJournal of Advertising, 2000
- Visual Rhetoric in Advertising: Text‐Interpretive, Experimental, and Reader‐Response AnalysesJournal of Consumer Research, 1999
- Emotion, Metaphor, and the Creative ProcessCreativity Research Journal, 1997
- Images in Advertising: The Need for a Theory of Visual RhetoricJournal of Consumer Research, 1994