UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training
Authors:
- 1 June 2021
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 4153-4163
Abstract
Vision-and-language pre-training has achieved impressive success in learning multimodal representations between vision and language. To generalize this success to non-English languages, we introduce UC 2 , the first machine translation-augmented framework for cross-lingual cross-modal representation learning. To tackle the scarcity problem of multilingual captions for image datasets, we first augment existing English-only datasets with other languages via machine translation (MT). Then we extend the standard Masked Language Modeling and Image-Text Matching training objectives to multilingual setting, where alignment between different languages is captured through shared visual context (i.e., using image as pivot). To facilitate the learning of a joint embedding space of images and all languages of interest, we further propose two novel pre-training tasks, namely Masked Region-to-Token Modeling (MRTM) and Visual Translation Language Modeling (VTLM), leveraging MT-enhanced translated data. Evaluation on multilingual image-text retrieval and multilingual visual question answering benchmarks demonstrates that our proposed framework achieves new state of the art on diverse non-English benchmarks while maintaining comparable performance to monolingual pre-trained models on English tasks.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Sentence-Level Multilingual Multi-modal Embedding for Natural Language Processing2017
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Doubly-Attentive Decoder for Multi-modal Neural Machine Translation2017
- STAIR Captions: Constructing a Large-Scale Japanese Image Caption Dataset2017
- Image Pivoting for Learning Multilingual Multimodal Representations2017
- Incorporating Global Visual Features into Attention-based Neural Machine Translation.2017
- CUNI System for the WMT17 Multimodal Translation Task2017
- Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description2017
- Multi30K: Multilingual English-German Image Descriptions2016
- Bridge Correlational Neural Networks for Multilingual Multimodal Representation Learning2016