Contrastive Learning with Soft Negative Sampling for Visual Word Sense Disambiguation
Open Access
- 7 July 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 180-183
Abstract
The task of visual word sense disambiguation (VWSD) is to find the images that accurately match target words or phrases from a given candidate set of images. It is a challenging problem, requiring VWSD models to understand the related information between images and texts and distinguish the ambiguous information. To address this problem, we propose a method of contrastive learning with soft negative sampling (CLSNS), which facilitates the optimization of VWSD models via comparisons of positive and soft negative samples. When calculating the text and image comparison loss, the ambiguous target word is used as the soft negative sample, the target phrase with known semantics is used as the positive sample, and other irrelevant text image pairs are used as negative samples. bidirectional margin loss (BML) is utilized to distinguish between negative samples and soft negative samples, and Info Noise-contrastive estimation (InfoNCE) loss to distinguish between positive and negative samples. To make the model could be trained more effectively, we expand the target word and target phrase using prompt technology. We can broaden the diversity of the training data by inserting various prompts which will enhance the model's functionality and generalizability. These augmentations can improve the model's comprehension of and ability to provide domain-specific content, resulting in more precise and focused responses. Moreover, an extended Wikipedia-based text model is introduced to enrich the input text information. Extensive experiments conducted on a VWSD dataset shared from SemEval 2023 demonstrate the effect of the proposed method.Keywords
This publication has 4 references indexed in Scilit:
- Image-text Retrieval: A Survey on Recent Research and Development2022
- Contextual sentiment embeddings via bi-directional GRU language modelKnowledge-Based Systems, 2022
- Learning sentiment sentence representation with multiview attention modelInformation Sciences, 2021
- On the Limitations of Visual-Semantic Embedding Networks for Image-to-Text Information RetrievalJournal of Imaging, 2021