DARE: Distill and Reinforce Ensemble Neural Networks for Climate-Domain Processing
Open Access
Abstract
Natural-language processing is well positioned to help stakeholders study the dynamics of ambiguous Climate Change-related (CC) information. Recently, deep neural networks have achieved good results on a variety of NLP tasks depending on high-quality training data and complex and exquisite frameworks. This raises two dilemmas: (1) the networks are highly reliant on powerful hardware devices and processing is time-consuming, which is not only inconducive to execution on edge devices but also leads to resource consumption. (2) Obtaining large-scale effective annotated data is difficult and laborious, especially when it comes to a special domain such as CC. In this paper, we propose a CC-domain-adapted BERT distillation and reinforcement ensemble (DARE) model for tackling the problems above. Specifically, we propose a novel data-augmentation strategy which is a Generator-Reinforced Selector collaboration network for countering the dilemma of CC-related data scarcity. Extensive experimental results demonstrate that our proposed method outperforms baselines with a maximum of 26.83% on SoTA and 50.65× inference time speed-up. Furthermore, as a remedy for the lack of CC-related analysis in the NLP community, we also provide some interpretable conclusions for this global concern.Keywords
This publication has 25 references indexed in Scilit:
- Does BERT need domain adaptation for clinical negation detection?Journal of the American Medical Informatics Association, 2020
- Pre-Training BERT on Domain Resources for Short Answer Grading2019
- Domain Adaptation with BERT-based Domain Classification and Data Selection2019
- Patient Knowledge Distillation for BERT Model Compression2019
- A Text Sentiment Classification Modeling Method Based on Coordinated CNN‐LSTM‐Attention ModelChinese Journal of Electronics, 2019
- Comparing Attitudes to Climate Change in the Media using sentiment analysis based on Latent Dirichlet Allocation2017
- Topically-focused Blog Corpora for Multiple Languages2016
- SemEval-2016 Task 6: Detecting Stance in Tweets2016
- Climate Change Sentiment on Twitter: An Unsolicited Public Opinion PollPLOS ONE, 2015
- Long Short-Term MemoryNeural Computation, 1997