What is sentiment meant to mean to language models?
Open Access
- 1 October 2024
- journal article
- research article
- Published by SAGE Publications in Research & Politics
- Vol. 11 (4)
Abstract
Sentiment analysis is one of the most widely used text analysis methods in social science. Recent advancements with large language models have made it more accurate and accessible than ever, allowing researchers to classify text with only a plain English prompt. However, “sentiment” entails a wide variety of concepts depending on the domain and tools used. It has been used to mean emotion, opinions, market movements, or simply a general “good-bad” dimension. This raises a question: What exactly are language models doing when prompted to label documents by sentiment? This paper first overviews how sentiment is defined across different contexts, highlighting that it is a confounded measurement construct that entails multiple concepts, particularly emotional valence and opinion, without disentangling them. I then test three language models across two data sets with prompts requesting sentiment, valence, and stance classification. I find that language models generally understand sentiment in terms of emotional valence rather than opinion. I find that classification generally improves when researchers more precisely specify their dimension, but that even when classification metrics are similar the different prompts are biased in different ways. I conclude by encouraging researchers to move beyond sentiment when feasible and use more precise measurement constructs.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- Stance detection: a practical guide to classifying political beliefs in textPolitical Science Research and Methods, 2024
- GPT is an effective tool for multilingual psychological text analysisProceedings of the National Academy of Sciences, 2024
- Persistent interaction patterns across social media platforms and over timeNature, 2024
- Sentiment is Not Stance: Target-Aware Opinion Classification for Political Text AnalysisPolitical Analysis, 2022
- A survey on sentiment analysis methods, applications, and challengesArtificial Intelligence Review, 2022
- Stance detection on social media: State of the art and trendsInformation Processing & Management, 2021
- Partisan Polarization Is the Primary Psychological Motivation behind Political Fake News Sharing on TwitterAmerican Political Science Review, 2021
- Using sentiment analysis to predict opinion inversion in Tweets of political communicationScientific Reports, 2021
- Affective News: The Automated Coding of Sentiment in Political TextsPolitical Communication, 2012
- Adverbial stance types in EnglishDiscourse Processes, 1988