A Visually Enhanced Neural Encoder for Synset Induction
Open Access
- 20 August 2023
- journal article
- research article
- Published by MDPI AG in Electronics
- Vol. 12 (16) , 3521
Abstract
The synset induction task is to automatically cluster semantically identical instances, which are often represented by texts and images. Previous works mainly consider textual parts, while ignoring the visual counterparts. However, how to effectively employ the visual information to enhance the semantic representation for the synset induction is challenging. In this paper, we propose a Visually Enhanced NeUral Encoder (i.e., VENUE) to learn a multimodal representation for the synset induction task. The key insight lies in how to construct multimodal representations through intra-modal and inter-modal interactions among images and text. Specifically, we first design the visual interaction module through the attention mechanism to capture the correlation among images. To obtain the multi-granularity textual representations, we fuse the pre-trained tags and word embeddings. Second, we design a masking module to filter out weakly relevant visual information. Third, we present a gating module to adaptively regulate the modalities’ contributions to semantics. A triplet loss is adopted to train the VENUE encoder for learning discriminative multimodal representations. Then, we perform clustering algorithms on the obtained representations to induce synsets. To verify our approach, we collect a multimodal dataset, i.e., MMAI-Synset, and conduct extensive experiments. The experimental results demonstrate that our method outperforms strong baselines on three groups of evaluation metrics.Keywords
Funding Information
- National Natural Science Foundation of China (62076032)
This publication has 32 references indexed in Scilit:
- Multi-Modal Word Synset Induction2017
- Partially tagged image clustering2015
- FaceNet: A unified embedding for face recognition and clustering2015
- From image-level to pixel-level labeling with Convolutional Networks2015
- Show and tell: A neural image caption generator2015
- Glove: Global Vectors for Word Representation2014
- Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics2014
- Maps of random walks on complex networks reveal community structureProceedings of the National Academy of Sciences, 2008
- On Clustering Validation TechniquesJournal of Intelligent Information Systems, 2001
- Mining the Web for Synonyms: PMI-IR versus LSA on TOEFL2001