Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation
Open Access
- 22 September 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 414-418
Abstract
The process of transforming input images into corresponding textual explanations stands as a crucial and complex endeavor within the domains of computer vision and natural language processing. In this paper, we propose an innovative ensemble approach that harnesses the capabilities of Contrastive Language-Image Pretraining (CLIP) models. Our ensemble framework encompasses two significant variations of the CLIP model, each meticulously designed to cater to specific nuances within the image-to-text transformation landscape. The first model introduces an elaborated architecture, featuring multiple layers with distinct learning rates, thereby amplifying its adeptness in capturing intricate relationships between images and text. The second model strategically exploits CLIP’s inherent zero-shot learning potential to generate image-text embeddings, subsequently harnessed by a K-Nearest Neighbors (KNN) model. Through this KNN-based paradigm, the model facilitates image-to-text transformation by identifying closely related embeddings within the embedding space. Notably, our ensemble approach is rigorously evaluated, employing the cosine similarity metric to gauge the alignment between model-generated embeddings and ground truth representations. Comparative experiments vividly highlight the superiority of our ensemble strategy over standalone CLIP models. This study not only advances the state-of-the-art in image-to-text transformation but also accentuates the promising trajectory of ensemble learning in effectively addressing intricate multimodal tasks.Keywords
All Related Versions
This publication has 10 references indexed in Scilit:
- Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet2021
- Stack-Captioning: Coarse-to-Fine Learning for Image CaptioningProceedings of the AAAI Conference on Artificial Intelligence, 2018
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning2018
- Self-Critical Sequence Training for Image Captioning2017
- Zero-Shot Learning — The Good, the Bad and the Ugly2017
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning2016
- Deep visual-semantic alignments for generating image descriptions2015
- CIDEr: Consensus-based image description evaluation2015
- Observations on Using Type-2 Fuzzy Logic for Reducing Semantic Gap in Content–Based Image Retrieval SystemInternational Journal of Computer Theory and Engineering, 2014
- K-nearest neighborScholarpedia, 2009