Bridging Human and Model Perspectives: A Comparative Analysis of Political Bias Detection in News Media Using Large Language Models
Authors:
- Niaz Nafi RahmanNRNiaz Nafi Rahman
- Tahsina MoiukhTMTahsina Moiukh
- Farig SadequeFSFarig Sadeque
Preprint
Abstract
Detecting political bias in news media is a complex task that requires interpreting subtle linguistic and contextual cues. Although recent advances in Natural Language Processing (NLP) have enabled automatic bias classification, the extent to which large language models (LLMs) align with human judgment still remains relatively underexplored and not yet well understood. This study aims to present a comparative framework for evaluating the detection of political bias across human annotations and multiple LLMs, including GPT, BERT, RoBERTa, and FLAN. We construct a manually annotated dataset of news articles and assess annotation consistency, bias polarity, and inter-model agreement to quantify divergence between human and model perceptions of bias. Experimental results show that among traditional transformer-based models, RoBERTa achieves the highest alignment with human labels, whereas generative models such as GPT demonstrate the strongest overall agreement with human annotations in a zero-shot setting. Among all transformer-based baselines, our fine-tuned RoBERTa model acquired the highest accuracy and the strongest alignment with human-annotated labels. Our findings highlight systematic differences in how humans and LLMs perceive political slant, underscoring the need for hybrid evaluation frameworks that combine human interpretability with model scalability in automated media bias detection.Keywords
All Related Versions
This publication has 15 references indexed in Scilit:
- Comparing large Language models and human annotators in latent content analysis of sentiment, political leaning, emotional intensity and sarcasmScientific Reports, 2025
- The FIGNEWS Shared Task on News Media Narratives2024
- A systematic review on media bias detection: What is media bias, how it is expressed, and how to detect itExpert Systems with Applications, 2023
- ChatGPT outperforms crowd workers for text-annotation tasksProceedings of the National Academy of Sciences of the United States of America, 2023
- Can Large Language Models Be an Alternative to Human Evaluations?2023
- From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models2023
- Dealing with Disagreements: Looking Beyond the Majority Vote in Subjective AnnotationsTransactions of the Association for Computational Linguistics, 2022
- The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation2022
- Neural Media Bias Detection Using Distant Supervision With BABE - Bias Annotations By Experts2021
- We Can Detect Your Bias: Predicting the Political Ideology of News Articles2020