Nonword-to-Image Generation Considering Perceptual Association of Phonetically Similar Words
Open Access
- 29 October 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 115-125
Abstract
Text-to-Image (T2I) generation has long been a popular field of multimedia processing. Recent advances in large-scale vision and language pretraining have brought a number of models capable of very high-quality T2I generation. However, they are reported to generate unexpected images when users input words that have no definition within a language (nonwords), including coined words and pseudo-words. To make the behavior of T2I generation models against nonwords more intuitive, we propose a method that considers phonetic information of text inputs. The phonetic similarity is adopted so that the generated images from a nonword contain the concept of its phonetically similar words. This is based on the psycholinguistic finding that humans would also associate nonwords with their phonetically similar words when they perceive the sound. Our evaluations confirm a better agreement of the generated images of the proposed method with both phonetic relationships and human expectations than a conventional T2I generation model. The cross-lingual comparison of generated images for a nonword highlights the differences in language-specific nonword-imagery correspondences. These results provide insight into the usefulness of the proposed method in brand naming and language learning.Keywords
Funding Information
- Japan Society for the Promotion of Science (22H03612)
This publication has 16 references indexed in Scilit:
- Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works2023
- ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP Cues2022
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Wav2CLIP: Learning Robust Audio Representations from Clip2022
- Generating Images from Caption and Vice Versa via CLIP-Guided Generative Latent Space Search2021
- What do phone embeddings learn about Phonology?2019
- Neural Machine Translation of Rare Words with Subword Units2016
- Sound Symbolism1995
- Form-based priming in spoken word recognition: The roles of competition and bias.Journal of Experimental Psychology: Learning, Memory, and Cognition, 1992
- Facilitation in recognizing pairs of words: Evidence of a dependence between retrieval operations.Journal of Experimental Psychology, 1971