DSReg: Using Distant Supervision as a Regularizer
Preprint
Abstract
In this paper, we aim at tackling a general issue in NLP tasks where some of the negative examples are highly similar to the positive examples, i.e., hard-negative examples. We propose the distant supervision as a regularizer (DSReg) approach to tackle this issue. The original task is converted to a multi-task learning problem, in which distant supervision is used to retrieve hard-negative examples. The obtained hard-negative examples are then used as a regularizer. The original target objective of distinguishing positive examples from negative examples is jointly optimized with the auxiliary task objective of distinguishing softened positive (i.e., hard-negative examples plus positive examples) from easy-negative examples. In the neural context, this can be done by outputting the same representation from the last neural layer to different $softmax$ functions. Using this strategy, we can improve the performance of baseline models in a range of different NLP tasks, including text classification, sequence labeling and reading comprehension.
Keywords
All Related Versions
This publication has 3 references indexed in Scilit:
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- The NarrativeQA Reading Comprehension ChallengeTransactions of the Association for Computational Linguistics, 2018
- Visualizing and Understanding Neural Models in NLP2016