MetaCLUE: Towards Comprehensive Visual Metaphors Research

Authors:
  • AAArjun R. Akula
    Arjun R. Akula
  • BDBrendan Driscoll
    Brendan Driscoll
  • PNPradyumna Narayana
    Pradyumna Narayana
  • SCSoravit Changpinyo
    Soravit Changpinyo
  • ZJZhiwei Jia
    Zhiwei Jia
  • SDSuyash Damle
    Suyash Damle
  • GPGarima Pruthi
    Garima Pruthi
  • SBSugato Basu
    Sugato Basu
  • LGLeonidas Guibas
    Leonidas Guibas
  • WFWilliam T. Freeman
    William T. Freeman
  • YLYuanzhen Li
    Yuanzhen Li
  • VJVarun Jampani
    Varun Jampani

Abstract
Creativity is an indispensable part of human cognition and also an inherent part of how we make sense of the world. Metaphorical abstraction is fundamental in communicating creative ideas through nuanced relationships between abstract concepts such as feelings. While computer vision benchmarks and approaches predominantly focus on understanding and generating literal interpretations of images, metaphorical comprehension of images remains relatively unexplored. Towards this goal, we introduce MetaCLUE, a set of vision tasks on visual metaphor. We also collect high-quality and rich metaphor annotations (abstract objects, concepts, relationships along with their corresponding object boxes) as there do not exist any datasets that facilitate the evaluation of these tasks. We perform a comprehensive analysis of state-of-the-art models in vision and language based on our annotations, highlighting strengths and weaknesses of current approaches in visual metaphor Classification, Localization, Understanding (retrieval, question answering, captioning) and gEneration (text-to-image synthesis) tasks. We hope this work provides a concrete step towards developing AI systems with human-like creative capabilities.

This publication has 8 references indexed in Scilit: