Adaptive Pre-Training and Collaborative Fine-Tuning: A Win-Win Strategy to Improve Review Analysis Tasks

Authors:
Publisher website
Google Scholar
Abstract
Summarizing user reviews and classifying user sentiment are two critical tasks for modern e-commerce platforms. These two tasks can benefit each other by capturing the shared linguistic features. However, such a relationship has not been fully exploited by existing research on domain-specific contextual representations. This work explores a win-win strategy for a multi-task framework with three stages: general pre-training, adaptive pre-training, and collaborative fine-tuning. The task-adaptive continual pre-training on a language model can obtain domain-specific contextual representations, further used to improve two related tasks, sentiment classification and review summarization during the collaborative fine-tuning. Meanwhile, to effectively capture sentiment-oriented domain-specific contextual representations, we introduce a novel task-adaptive pre-training procedure, which adds a sentiment prediction task during the adaptive pre-training. Extensive experiments conducted on two adaption scenarios of a general-to-single domain and a general-to-multiple domain show that our framework outperforms state-of-the-art methods.
Funding Information
  • NSF (III-1763325, III-1909323, III-2106758, SaTC-1930941)
  • National Natural Science Foundation of China (61872022)
  • Engineering and Physical Sciences Research Council (EP/P011829/1)
  • Academic Excellence Foundation of BUAA for PHD Students