A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned, and Perspectives
Open Access
- 2 February 2023
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Computing Surveys
- Vol. 55 (10) , 1-17
Abstract
Modern natural language processing (NLP) methods employ self-supervised pretraining objectives such as masked language modeling to boost the performance of various downstream tasks. These pretraining methods are frequently extended with recurrence, adversarial, or linguistic property masking. Recently, contrastive self-supervised training objectives have enabled successes in image representation pretraining by learning to contrast input-input pairs of augmented images as either similar or dissimilar. In NLP however, a single token augmentation can invert the meaning of a sentence during input-input contrastive learning, which led to input-output contrastive approaches that avoid the issue by instead contrasting over input-label pairs. In this primer, we summarize recent self-supervised and supervised contrastive NLP pretraining methods and describe where they are used to improve language modeling, zero to few-shot learning, pretraining data-efficiency, and specific NLP tasks. We overview key contrastive learning concepts with lessons learned from prior research and structure works by applications. Finally, we point to open challenges and future directions for contrastive NLP to encourage bringing contrastive NLP pretraining closer to recent successes in image representation pretraining.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- CERT: Contrastive Self-supervised Learning for Language UnderstandingTechRxiv, 2020
- Words Aren’t Enough, Their Order Matters: On the Robustness of Grounding Visual Referring Expressions2020
- SPECTER: Document-level Representation Learning using Citation-informed Transformers2020
- Pretraining with Contrastive Sentence Objectives Improves Discourse Performance of Language Models2020
- MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classification2020
- Contrastive Self-Supervised Learning for Commonsense Reasoning2020
- Transferable Contrastive Network for Generalized Zero-Shot Learning2019
- Contrastive Attention Mechanism for Abstractive Sentence Summarization2019
- The MuCoW Test Suite at WMT 2019: Automatically Harvested Multilingual Contrastive Word Sense Disambiguation Test Sets for Machine Translation2019
- Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency2018