Fine-Tuning Transformer Models Using Transfer Learning for Multilingual Threatening Text Identification
Open Access
- 27 September 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 106503-106515
Abstract
Threatening content detection on social media has recently gained attention. There is very limited work regarding threatening content detection in low-resource languages, especially in Urdu. Furthermore, previous work explored only mono-lingual approaches, and multi-lingual threatening content detection was not studied. This research addressed the task of Multi-lingual Threatening Content Detection (MTCD) in Urdu and English languages by exploiting transfer learning methodology with fine-tuning techniques. To address the multi-lingual task, we investigated two methodologies: 1) Joint multi-lingual, and 2) Joint-translated method. The former approach employs the concept of building a universal classifier for different languages whereas the latter approach applies the translation process to transform the text into one language and then perform classification. We explore the Multilingual Representations for Indian Languages (MuRIL) and Robustly Optimized BERT Pre-Training Approach (RoBERTa) with fine-tuning that already demonstrated state-of-the-art in capturing the contextual and semantic characteristics within the text. For hyper-parameters, manual search and grid search strategies are utilized to find the optimum values. Various experiments are performed on bi-lingual English and Urdu datasets and findings revealed that the proposed methodology outperformed the baselines and showed benchmark performance. The RoBERTa model achieved the highest performance by demonstrating 92% accuracy and 90% macro f1-score with the joint multi-lingual approach.Keywords
Funding Information
- Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia (PNURSP2023R104)
- Computational Resources of High Performance Computing (HPC) Facilities at National Research University Higher School of Economics
This publication has 20 references indexed in Scilit:
- Deep-BERT: Transfer Learning for Classifying Multilingual Offensive Texts on Social MediaComputer Systems Science and Engineering, 2023
- A multilingual offensive language detection method based on transfer learning from transformer fine-tuning modelJournal of King Saud University Computer and Information Sciences, 2022
- Threatening Language Detection and Target Identification in Urdu TweetsIEEE Access, 2021
- Individual vs. Group Violent Threats Classification in Online Discussions2020
- THREAT: A Large Annotated Corpus for Detection of Violent Threats2019
- Bidirectional LSTM with attention mechanism and convolutional layer for text classificationNeurocomputing, 2019
- Threat detection in online discussions2016
- Shallow parsing for recognizing threats in Dutch tweets2013
- N-Gram-Based Recognition of Threatening Tweets2013
- Combining coregularization and consensus-based self-training for multilingual text categorization2010