ZeroNLG: Aligning and Autoencoding Domains for Zero-Shot Multimodal and Multilingual Natural Language Generation
- 29 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 46 (8) , 5712-5724
Abstract
Natural Language Generation (NLG) accepts input data in the form of images, videos, or text and generates corresponding natural language text as output. Existing NLG methods mainly adopt a supervised approach and rely heavily on coupled data-to-text pairs. However, for many targeted scenarios and for non-English languages, sufficient quantities of labeled data are often not available. As a result, it is necessary to collect and label data-text pairs for training, which is both costly and time-consuming. To relax the dependency on labeled data of downstream tasks, we propose an intuitive and effective zero-shot learning framework, ZeroNLG, which can deal with multiple NLG tasks, including image-to-text (image captioning), video-to-text (video captioning), and text-to-text (neural machine translation), across English, Chinese, German, and French within a unified framework. ZeroNLG does not require any labeled downstream pairs for training. During training, ZeroNLG (i) projects different domains (across modalities and languages) to corresponding coordinates in a shared common latent space; (ii) bridges different domains by aligning their corresponding coordinates in this space; and (iii) builds an unsupervised multilingual auto-encoder to learn to generate text by reconstructing the input text given its coordinate in shared latent space. Consequently, during inference, based on the data-to-text pipeline, ZeroNLG can generate target sentences across different languages given the coordinate of input data in the common space. Within this unified framework, given visual (imaging or video) data as input, ZeroNLG can perform zero-shot visual captioning; given textual sentences as input, ZeroNLG can perform zero-shot machine translation. We present the results of extensive experiments on twelve NLG tasks, showing that, without using any labeled downstream pairs for training, ZeroNLG generates high-quality and “believable” outputs and significantly outperforms existing zero-shot methods. Our code and data are available at https://github.com/yangbang18/ZeroNLG .Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62176008)
- Shenzhen Science and Technology Research Program (GXWD20201231165807007-20200814115301001)
- Peng Cheng Laboratory (PCL2023A08)
- University of Oxford
- National Institute for Health Research
- NIHR Research Professorship
- Royal Academy of Engineering Research Chair
- Well-come Trust
- U.K. Research and Innovation
- Engineering and Physical Sciences Research Council
- InnoHK Hong Kong Centre for Cerebro-cardiovascular Engineering
- Clarendon Fund
- Magdalen Graduate Scholarship
This publication has 56 references indexed in Scilit:
- Rethinking Few-Shot Image Classification: A Good Embedding is All You Need?2020
- Meshed-Memory Transformer for Image Captioning2020
- Unsupervised Image Captioning2019
- Unpaired Image Captioning by Language Pivoting2018
- Neural Motifs: Scene Graph Parsing with Global Context2018
- Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors2017
- SPICE: Semantic Propositional Image Caption Evaluation2016
- MSR-VTT: A Large Video Description Dataset for Bridging Video and Language2016
- Show and tell: A neural image caption generator2015
- CIDEr: Consensus-based image description evaluation2015