Text-to-Image Generation for Abstract Concepts
Top Cited Papers
Open Access
- 24 March 2024
- journal article
- research article
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 38 (4) , 3360-3368
Abstract
Recent years have witnessed the substantial progress of large-scale models across various domains, such as natural language processing and computer vision, facilitating the expression of concrete concepts. Unlike concrete concepts that are usually directly associated with physical objects, expressing abstract concepts through natural language requires considerable effort since they are characterized by intricate semantics and connotations. An alternative approach is to leverage images to convey rich visual information as a supplement. Nevertheless, existing Text-to-Image (T2I) models are primarily trained on concrete physical objects and often struggle to visualize abstract concepts. Inspired by the three-layer artwork theory that identifies critical factors, intent, object and form during artistic creation, we propose a framework of Text-to-Image generation for Abstract Concepts (TIAC). The abstract concept is clarified into a clear intent with a detailed definition to avoid ambiguity. LLMs then transform it into semantic-related physical objects, and the concept-dependent form is retrieved from an LLM-extracted form pattern set. Information from these three aspects will be integrated to generate prompts for T2I models via LLM. Evaluation results from human assessments and our newly designed metric concept score demonstrate the effectiveness of our framework in creating images that can sufficiently express abstract concepts.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- A taxonomy of prompt modifiers for text-to-image generationBehaviour & Information Technology, 2023
- ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation2023
- DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation2023
- MetaCLUE: Towards Comprehensive Visual Metaphors Research2023
- Multi-Concept Customization of Text-to-Image Diffusion2023
- MET-Meme: A Multimodal Meme Dataset Rich in Metaphors2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks2018
- Automatic Understanding of Image and Video Advertisements2017
- Automated Flower Classification over a Large Number of Classes2008