LLM Teacher-Student Framework for Text Classification With No Manually Annotated Data: A Case Study in IPTC News Topic Classification
Open Access
- 24 February 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 35621-35633
Abstract
With the ever-increasing number of news stories available online, classifying them by topic, regardless of the language they are written in, has become crucial for enhancing readers’ access to relevant content. To address this challenge, we propose a teacher-student framework based on large language models (LLMs) for developing multilingual news topic classification models of reasonable size with no need for manual data annotation. The framework employs a Generative Pretrained Transformer (GPT) model as the teacher model to develop a news topic training dataset through automatic annotation of 20,000 news articles in Slovenian, Croatian, Greek, and Catalan. Articles are classified into 17 main categories from the Media Topic schema, developed by the International Press Telecommunications Council (IPTC). The teacher model exhibits a high zero-shot performance on all four languages. Its agreement with human annotators is comparable to that between the human annotators themselves. To mitigate the computational limitations associated with the requirement of processing millions of texts daily, smaller BERT-like student models are fine-tuned on the GPT-annotated dataset. These student models achieve high performance comparable to the teacher model. Furthermore, we explore the impact of the training data size on the performance of the student models and investigate their monolingual, multilingual and zero-shot cross-lingual capabilities. The findings indicate that student models can achieve high performance with a relatively small number of training instances, and demonstrate strong zero-shot cross-lingual abilities. Finally, we publish the best-performing news topic classifier, enabling multilingual classification with the top-level categories of the IPTC Media Topic schema.Keywords
All Related Versions
Funding Information
- Projects “Embeddings-Based Techniques for Media Monitoring Applications” co-funded by the Kliping d.o.o. Agency and the Slovenian Research and Innovation Agency (L2-50070)
- “Large Language Models for Digital Humanities” funded by ARIS (GC-0002)
- Research Programme “Language Resources and Technologies for Slovene” funded by ARIS (P6-0411)
This publication has 29 references indexed in Scilit:
- Analyzing Dataset Annotation Quality Management in the WildComputational Linguistics, 2024
- Evaluating the Effectiveness of GPT Large Language Model for News Classification in the IPTC News OntologyIEEE Access, 2023
- Automatic genre identification: a surveyLanguage Resources and Evaluation, 2023
- Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech2023
- EventDNA: a dataset for Dutch news event extraction as a basis for news diversificationLanguage Resources and Evaluation, 2022
- The Validity of Sentiment Analysis: Comparing Manual Annotation, Crowd-Coding, Dictionary Approaches, and Machine Learning AlgorithmsCommunication Methods and Measures, 2021
- The FRENK Datasets of Socially Unacceptable Discourse in Slovene and English2019
- Content Analysis by the Crowd: Assessing the Usability of Crowdsourcing for Coding Latent ConstructsCommunication Methods and Measures, 2017
- Bangla news classification using naive Bayes classifier2014
- Measuring the Reliability of Qualitative Text Analysis DataQuality & Quantity, 2004