What is Sentiment Meant to Mean to Language Models?
Preprint
Abstract
Sentiment analysis is one of the most widely used techniques in text analysis. Recent advancements with Large Language Models have made it more accurate and accessible than ever, allowing researchers to classify text with only a plain English prompt. However, "sentiment" entails a wide variety of concepts depending on the domain and tools used. It has been used to mean emotion, opinions, market movements, or simply a general ``good-bad'' dimension. This raises a question: What exactly are language models doing when prompted to label documents by sentiment? This paper first overviews how sentiment is defined across different contexts, highlighting that it is a confounded measurement construct in that it entails multiple variables, such as emotional valence and opinion, without disentangling them. I then test three language models across two data sets with prompts requesting sentiment, valence, and stance classification. I find that sentiment labels most strongly correlate with valence labels. I further find that classification improves when researchers more precisely specify their dimension of interest rather than using the less well-defined concept of sentiment. I conclude by encouraging researchers to move beyond "sentiment" when feasible and use a more precise measurement construct.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- GPT is an effective tool for multilingual psychological text analysisProceedings of the National Academy of Sciences, 2024
- The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classificationBioData Mining, 2023
- Sentiment is Not Stance: Target-Aware Opinion Classification for Political Text AnalysisPolitical Analysis, 2022
- Stance detection on social media: State of the art and trendsInformation Processing & Management, 2021
- Partisan Polarization Is the Primary Psychological Motivation behind Political Fake News Sharing on TwitterAmerican Political Science Review, 2021
- Using sentiment analysis to predict opinion inversion in Tweets of political communicationScientific Reports, 2021
- Your Stance is Exposed! Analysing Possible Factors for Stance Detection on Social MediaProceedings of the ACM on Human-Computer Interaction, 2019
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding2019
- Affective News: The Automated Coding of Sentiment in Political TextsPolitical Communication, 2012
- Opinion Mining and Sentiment Analysis2008