Focused Contrastive Loss for Classification With Pre-Trained Language Models
- 3 November 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Knowledge and Data Engineering
- Vol. 36 (7) , 3047-3061
Abstract
Contrastive learning, which learns data representations by contrasting similar and dissimilar instances, has achieved great success in various domains including natural language processing (NLP). Recently, it has been demonstrated that incorporating class labels into contrastive learning, i.e., supervised contrastive learning (SCL), can further enhance the quality of the learned data representations. Although several works have shown empirically that incorporating SCL into classification models leads to better performance, the mechanism of how SCL works for classification is less studied. In this paper, we first investigate how SCL facilitates the classifier learning, where we show that the contrastive region, i.e., the data instances involved in each contrasting operation, has a crucial link to the mechanism of SCL. We reveal that the vanilla SCL is suboptimal since its behavior can be altered by variances in class distributions. Based on this finding, we propose a Fo cused C ontrastive L oss (FoCL) for classification. Compared with SCL, FoCL defines a finer contrastive region, focusing on the data instances surrounding decision boundaries. We conduct extensive experiments on three NLP tasks: text classification, named entity recognition, and relation extraction. Experimental results show consistent and significant improvements of FoCL over strong baselines on various benchmark datasets, especially in few-shot scenarios.Keywords
Funding Information
- Australian Research Council Linkage Project (LP160101469)
- Elsevier
This publication has 43 references indexed in Scilit:
- Fighting Class Imbalance with Contrastive Learning2021
- Contrastive Multiview Coding2020
- A Unifying Mutual Information View of Metric Learning: Cross-Entropy vs. Pairwise Losses2020
- GPT-3: Its Nature, Scope, Limits, and ConsequencesMinds and Machines, 2020
- Overview of ChEMU 2020: Named Entity Recognition and Event Extraction of Chemical Reactions from Patents2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- BioBERT: a pre-trained biomedical language representation model for biomedical text miningBioinformatics, 2019
- Sentiment analysis algorithms and applications: A surveyAin Shams Engineering Journal, 2014
- Cost-sensitive boosting for classification of imbalanced dataPattern Recognition, 2007
- A Tutorial on the Cross-Entropy MethodAnnals of Operations Research, 2005