What’s the Tone? Easy Doesn’t Do It: Analyzing Performance and Agreement Between Off-the-Shelf Sentiment Analysis Tools
Open Access
- 17 October 2019
- journal article
- research article
- Published by Taylor & Francis in Communication Methods and Measures
- Vol. 14 (2) , 83-104
Abstract
This article scrutinizes the method of automated content analysis to measure the tone of news coverage. We compare a range of off-the-shelf sentiment analysis tools to manually coded economic news as well as examine the agreement between these dictionary approaches themselves. We assess the performance of five off-the-shelf sentiment analysis tools and two tailor-made dictionary-based approaches. The analyses result in five conclusions. First, there is little overlap between the off-the-shelf tools; causing wide divergence in terms of tone measurement. Second, there is no stronger overlap with manual coding for short texts (i.e., headlines) than for long texts (i.e., full articles). Third, an approach that combines individual dictionaries achieves a comparably good performance. Fourth, precision may increase to acceptable levels at higher levels of granularity. Fifth, performance of dictionary approaches depends more on the number of relevant keywords in the dictionary than on the number of valenced words as such; a small tailor-made lexicon was not inferior to large established dictionaries. Altogether, we conclude that off-the-shelf sentiment analysis tools are mostly unreliable and unsuitable for research purposes – at least in the context of Dutch economic news – and manual validation for the specific language, domain, and genre of the research project at hand is always warranted.Keywords
Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (016.145.369)
This publication has 47 references indexed in Scilit:
- Norms of valence, arousal, dominance, and age of acquisition for 4,300 Dutch wordsBehavior Research Methods, 2012
- Affective News: The Automated Coding of Sentiment in Political TextsPolitical Communication, 2012
- Sentiment strength detection in short informal textJournal of the American Society for Information Science and Technology, 2010
- Good News or Bad News? Conducting Sentiment Analysis on Dutch Text to Distinguish Between Positive and Negative RelationsJournal of Information Technology & Politics, 2008
- Giving Content to Investor Sentiment: The Role of Media in the Stock MarketThe Journal of Finance, 2007
- Good News and Bad News: Asymmetric Responses to Economic InformationThe Journal of Politics, 2006
- Determining Economic News CoverageInternational Journal of Public Opinion Research, 2005
- The Conditioned Impact of Recession News: A Time-Series Analysis of Economic Communication in the United States, 1987-1996International Journal for Quality in Health Care, 2002
- Media Coverage of the Economy and Aggregate Economic Evaluations: Uncovering Evidence of Indirect Media EffectsPolitical Research Quarterly, 1995
- RECESSION HEADLINE NEWS, CONSUMER SENTIMENT, THE STATE OF THE ECONOMY AND PRESIDENTIAL POPULARITY: A TIME SERIES ANALYSIS 1989–1993International Journal of Public Opinion Research, 1995