Fine-tuning techniques and data augmentation on transformer-based models for conversational texts and noisy user-generated content
Authors:
- 7 December 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 919-925
Abstract
Transfer learning and Transformer-based language models play important roles in modern natural language processing research community. In this paper, we propose Transformer model's fine-tuning and data augmentation (TMFTDA) techniques for conversational texts and noisy user-generated content. We use two NTCIR-15 tasks, namely the first Dialogue Evaluation (DialEval-1) task and the second Numeral Attachment in Financial Tweets (FinNum-2) task, to evaluate the efficacy of TMFTDA. Experimental results show that TMFTDA substantially outperforms the baselines model of Bidirectional Long Short-Term Memory (Bi-LSTM) in multi-turn dialogue system evaluation at DialEval-1's Dialogue Quality (DQ) and Nugget Detection (ND) subtasks. Moreover, TMFTDA performs to a satisfactory level at FinNum-2 with a model of Cross-lingual Language Models using a Robustly Optimized BERT Pretraining Approach (XLM-RoBERTa). The research contribution of this paper is that, we help shed some light on the usefulness of TMFTDA, for conversational texts and noisy user-generated content in social media text analytics.Keywords
Funding Information
- Tamkang University (TKU)
- Ministry of Science and Technology (MOST) (109-2221-E-324-024)
This publication has 14 references indexed in Scilit:
- Final Report of the NTCIR-14 FinNum Task: Challenges and Current Status of Fine-Grained Numeral Understanding in Financial Social Media Data2019
- To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks2019
- SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing2018
- Universal Language Model Fine-tuning for Text Classification2018
- Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates2018
- Cyclical Learning Rates for Training Neural Networks2017
- Neural Machine Translation of Rare Words with Subword Units2016
- Glove: Global Vectors for Word Representation2014
- Framewise phoneme classification with bidirectional LSTM and other neural network architecturesNeural Networks, 2005
- Long Short-Term MemoryNeural Computation, 1997