Augmenting Sentiment Analysis with a Hierarchical Pre-Attention Framework and Strategic LLM Fine-Tuning

Authors:
Open Access
Publisher website
Google Scholar
Abstract
Sentiment analysis is significant for exploiting countless opinion-rich data from social media. However, it faces known challenges, such as informal expression and data sparsity. While large language models (LLMs) excel at dynamic contextual disambiguation, such deep learning models neglect to leverage prior knowledge calculated statistically and globally based on dataset-wide category tendencies to guide the encoding process. To overcome these challenges, this study proposes a pre-attention framework that includes a modified gated recurrent unit hierarchically integrated with a dual-level pre-attention mechanism (PA-GRU) and an LLM-PA-GRU model equipped with a strategic LLM fine-tuning method. The pre-attention mechanism extracts inter-category and intra-text statistical priors from the training set. Trainable coefficient matrices are proposed to adaptively fuse these priors within a unified formulation, enabling flexible allocation of global and local statistical signals. The proposed PA-GRU cell features a modified gating structure and a computation flow that integrates these extracted priors as explicit statistical anchors during sequence encoding. Moreover, we implement an LLM-PA-GRU model that connects the LLM’s deep feature representations to the PA-GRU. This strategic LLM fine-tuning approach is intended to mitigate gradient instability frequently encountered in models with heterogeneous architectures. Finally, a prototype-based alignment loss is employed to enforce feature consistency across modules. Extensive experiments on benchmark datasets demonstrate that the proposed approach achieves competitive performance compared with recent literature-reported models.
Funding Information
  • Research Base of Online Education for Shanghai Middle and Primary Schools

This publication has 39 references indexed in Scilit: