Transfer Fine-Tuning: A BERT Case Study
Preprint
Abstract
A semantic equivalence assessment is defined as a task that assesses semantic equivalence in a sentence pair by binary judgment (i.e., paraphrase identification) or grading (i.e., semantic textual similarity measurement). It constitutes a set of tasks crucial for research on natural language understanding. Recently, BERT realized a breakthrough in sentence representation learning (Devlin et al., 2019), which is broadly transferable to various NLP tasks. While BERT's performance improves by increasing its model size, the required computational power is an obstacle preventing practical applications from adopting the technology. Herein, we propose to inject phrasal paraphrase relations into BERT in order to generate suitable representations for semantic equivalence assessment instead of increasing the model size. Experiments on standard natural language understanding tasks confirm that our method effectively improves a smaller BERT model while maintaining the model size. The generated model exhibits superior performance compared to a larger BERT model on semantic equivalence assessment tasks. Furthermore, it achieves larger performance gains on tasks with limited training datasets for fine-tuning, which is a property desirable for transfer learning.Keywords
All Related Versions
This publication has 9 references indexed in Scilit:
- Deep Contextualized Word Representations2018
- A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference2018
- Enhanced LSTM for Natural Language Inference2017
- SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation2017
- A Continuously Growing Dataset of Sentential Paraphrases2017
- Supervised Learning of Universal Sentence Representations from Natural Language Inference Data2017
- SQuAD: 100,000+ Questions for Machine Comprehension of Text2016
- Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement2016
- Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks2015