When ‘YES’ Meets ‘but’: Can AI Comprehend Contradictory Humor in Comics?
- 29 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-15
Abstract
Understanding humor, especially when it involves complex and contradictory narratives requiring comparative reasoning ability, remains a significant challenge for large vision language models (VLMs). This limitation hinders AI's ability to engage in human-like reasoning and cultural expression. In this paper, we investigate this challenge through an in depth analysis of comics that juxtapose panels to create humor through contradictions. We introduce the YESBUT, a novel benchmark with 1,262 comic images from diverse multilingual and multicultural contexts, featuring comprehensive annotations that capture various aspects of narrative understanding. Using this benchmark, we systematically evaluate a wide range of VLMs through four complementary tasks spanning from surface content comprehension to deep narrative reasoning, with particular emphasis on comparative reasoning between contradictory elements. Our extensive experiments reveal that even the most advanced models significantly underperform compared to humans, with common failures in visual perception, key element identification, comparative analysis, and hallucinations. We further investigate text-based training strategies and social knowledge augmentation methods to enhance model performance. Our findings not only highlight critical weaknesses in VLMs' understanding of cultural and creative expressions but also provide pathways toward developing context-aware models capable of deeper narrative understanding through comparative reasoning.Keywords
Funding Information
- NSF
- High Performance Computing Resource in the Core Facility for Advanced Research Computing with Case Western Reserve University (NSF-2117439)
- OpenAI Researcher Access (0000007745)
This publication has 40 references indexed in Scilit:
- DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learningNature, 2025
- Improved Baselines with Visual Instruction Tuning2024
- SEED-Bench: Benchmarking Multimodal Large Language Models2024
- Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation2024
- Best humans still outperform artificial intelligence in a creative divergent thinking taskScientific Reports, 2023
- Visual Programming: Compositional visual reasoning without training2023
- Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality2022
- We are Humor Beings: Understanding and Predicting Visual Humor2016
- JUXTAPOSITION: A NEW WAY TO COMBINE LOGICSThe Review of Symbolic Logic, 2011
- Rethinking literacy: communication, representation and textReading, 2003