Do LLMs Track Public Opinion? A Multi-Model Study of Favorability Predictions in the 2024 U.S. Presidential Election
Preprint
Abstract
We investigate whether Large Language Models (LLMs) can track public opinion as measured by exit polls during the 2024 U.S. presidential election cycle. Our analysis focuses on headline favorability (e.g., "Favorable" vs. "Unfavorable") of presidential candidates across multiple LLMs queried daily throughout the election season. Using the publicly available llm-election-data-2024 dataset, we evaluate predictions from nine LLM configurations against a curated set of five high-quality polls from major organizations including Reuters, CNN, Gallup, Quinnipiac, and ABC. We find systematic directional miscalibration. For Kamala Harris, all models overpredict favorability by 10-40% relative to polls. For Donald Trump, biases are smaller (5-10%) and poll-dependent, with substantially lower cross-model variation. These deviations persist under temporal smoothing and are not corrected by internet-augmented retrieval. We conclude that off-the-shelf LLMs do not reliably track polls when queried in a straightforward manner and discuss implications for election forecasting.Keywords
All Related Versions
This publication has 12 references indexed in Scilit:
- LLM-generated messages can persuade humans on policy issuesNature Communications, 2025
- Synthesizing Public Opinions with LLMs: Role Creation, Impacts, and the Future of Edemorcacy2025
- Specializing Large Language Models to Simulate Survey Response Distributions for Global Populations2025
- Evaluating the persuasive influence of political microtargeting with large language modelsProceedings of the National Academy of Sciences, 2024
- Synthetic Replacements for Human Survey Data? The Perils of Large Language ModelsPolitical Analysis, 2024
- From News to Forecast: Integrating Event Analysis in LLM-Based Time Series Forecasting with Reflection2024
- Approaching Human-Level Forecasting with Language Models2024
- Hidden Persuaders: LLMs’ Political Leaning and Their Influence on Voters2024
- Virtual Personas for Language Models via an Anthology of Backstories2024
- An Evaluation of the 2016 Election Polls in the United StatesPublic Opinion Quarterly, 2018