Tibetan-BERT-wwm: A Tibetan Pretrained Model With Whole Word Masking for Text Classification
- 27 March 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 11 (5) , 6268-6277
Abstract
Social networks contributed massive text data generated by users in it, which were crucial in information explosion. These unstructured and ambiguous expressed data may result in difficulty in obtaining available contextual information from it, which can help us gain more accurate insights into user-generated content, user preferences, and topic dynamics within social networks. By training on a large-scale unsupervised corpus and fine-tuning parameters using a limited amount of supervised data, the pretrained language model can effectively capture rich contextual information and achieve excellent performance in numerous downstream tasks of natural language processing (NLP). For the low-resource language such as Tibetan, the distributed representation results of dynamic changes obtained from pretrained language models can effectively alleviate the problem of insufficient labeled data. In order to achieve more effective contextual information and word-level semantic information in Tibetan social media, we collected a large amount of Tibetan language corpus and trained a Tibetan pretrained language model, named as Tibetan-BERT-wwm, by using the whole word masking strategy. Additionally, we apply the model to analyze textual data from social networks to assess its efficacy in capturing user sentiments and news topic in Tibetan social media. In this study, accuracy, precision, recall, and F1 score were used to evaluate its performance. The results showed that the macro-F1 of Tibetan-BERT-wwm in the public dataset TNCC document and title are 75.55% and 64.17%, the self-built sentiment analysis dataset is 70.98%. Compared with other pretrained language models, the Tibetan-BERT-wwm model can capture the semantic information of Tibetan well and improve the Tibetan classification effect.Keywords
Funding Information
- National Key R&D Program of China (2020YFC0832500)
- National Natural Science Foundation of China (62266037)
- Science and technology plan of Qinghai Province (2020-GX-164)
This publication has 19 references indexed in Scilit:
- BERT-deep CNN: state of the art for sentiment analysis of COVID-19 tweetsSocial Network Analysis and Mining, 2023
- Tibert, n.2023
- Text classification based on machine learning for Tibetan social network2022
- Character-based Joint Word Segmentation and Part-of-Speech Tagging for Tibetan Based on Deep LearningACM Transactions on Asian and Low-Resource Language Information Processing, 2022
- Pre-Training With Whole Word Masking for Chinese BERTIEEE/ACM Transactions on Audio, Speech, and Language Processing, 2021
- Tibetan Text Classification Algorithm Based on Syllables2020
- Convolutional Recurrent Neural Networks for Text Classification2019
- End-to-End Neural Text Classification for Tibetan2017
- Deep Pyramid Convolutional Neural Networks for Text Categorization2017
- Cino2011