A Novel Pretrained General-purpose Vision Language Model for the Vietnamese Language
Open Access
- 10 May 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Asian and Low-Resource Language Information Processing
- Vol. 23 (5) , 1-16
Abstract
Lying in the cross-section of computer vision and natural language processing, vision language models are capable of processing images and text at once. These models are helpful in various tasks: text generation from image and vice versa, image-text retrieval, or visual navigation. Besides building a model trained on a dataset for a task, people also study general-purpose models to utilize many datasets for multitasks. Their two primary applications are image captioning and visual question answering. For English, large datasets and foundation models are already abundant. However, for Vietnamese, they are still limited. To expand the language range, this work proposes a pretrained general-purpose image-text model named VisualRoBERTa. A dataset of 600k images with captions (translated MS COCO 2017 from English to Vietnamese) is introduced to pretrain VisualRoBERTa. The model’s architecture is built using Convolutional Neural Network and Transformer blocks. Fine-tuning VisualRoBERTa shows promising results on the ViVQA dataset with 34.49% accuracy, 0.4173 BLEU 4, and 0.4390 RougeL (in visual question answering task), and best outcomes on the sViIC dataset with 0.6685 BLEU 4, 0.6320 RougeL (in image captioning task).Keywords
This publication has 30 references indexed in Scilit:
- Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and MethodsJournal of Artificial Intelligence Research, 2021
- UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training2021
- Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks2020
- UNITER: UNiversal Image-TExt Representation Learning2020
- Unified Vision-Language Pre-Training for Image Captioning and VQAProceedings of the AAAI Conference on Artificial Intelligence, 2020
- From Recognition to Cognition: Visual Commonsense Reasoning2019
- GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering2019
- Training Tips for the Transformer ModelThe Prague Bulletin of Mathematical Linguistics, 2018
- CIDEr: Consensus-based image description evaluation2015
- Microsoft COCO: Common Objects in Context2014