Pretrained Language Representations for Text Understanding: A Weakly-Supervised Perspective
Open Access
- 4 August 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 5817-5818
Abstract
Language representations pretrained on general-domain corpora and adapted to downstream task data have achieved enormous success in building natural language understanding (NLU) systems. While the standard supervised fine-tuning of pretrained language models (PLMs) has proven an effective approach for superior NLU performance, it often necessitates a large quantity of costly human-annotated training data. For example, the enormous success of ChatGPT and GPT-4 can be largely credited to their supervised fine-tuning with massive manually-labeled prompt-response training pairs. Unfortunately, obtaining large-scale human annotations is in general infeasible for most practitioners. To broaden the applicability of PLMs to various tasks and settings, weakly-supervised learning offers a promising direction to minimize the annotation requirements for PLM adaptions. In this tutorial, we cover the recent advancements in pretraining language models and adaptation methods for a wide range of NLU tasks. Our tutorial has a particular focus on weakly-supervised approaches that do not require massive human annotations. We will introduce the following topics in this tutorial: (1) pretraining language representation models that serve as the fundamentals for various NLU tasks, (2) extracting entities and hierarchical relations from unlabeled texts, (3) discovering topical structures from massive text corpora for text organization, and (4) understanding documents and sentences with weakly-supervised techniques.Keywords
Funding Information
- National Science Foundation (IIS-19-56151, IIS-17-41317, IIS 17-04532, 2019897, 2118329)
- Defense Advanced Research Projects Agency (FA8750-19-2-1004, HR001121C0165)
This publication has 17 references indexed in Scilit:
- Distantly-Supervised Named Entity Recognition with Noise-Robust Learning and Language Model Augmented Self-Training2021
- Weakly-supervised Text Classification Based on Keyword Graph2021
- TaxoClass: Hierarchical Multi-Label Text Classification Using Only Class Names2021
- X-Class: Text Classification with Extremely Weak Supervision2021
- CoRel: Seed-Guided Topical Taxonomy Construction by Concept Learning and Relation Transferring2020
- Hierarchical Topic Mining via Joint Spherical Tree and Text Embedding2020
- Minimally Supervised Categorization of Text with Metadata2020
- Discriminative Topic Mining via Category-Name Guided Text Embedding2020
- Text Classification Using Label Names Only: A Language Model Self-Training Approach2020
- META: Metadata-Empowered Weak Supervision for Text Classification2020