Assessing Generalization for Subpopulation Representative Modeling via In-Context Learning
Preprint
Abstract
This study evaluates the ability of Large Language Model (LLM)-based Subpopulation Representative Models (SRMs) to generalize from empirical data, utilizing in-context learning with data from the 2016 and 2020 American National Election Studies. We explore generalization across response variables and demographic subgroups. While conditioning with empirical data improves performance on the whole, the benefit of in-context learning varies considerably across demographics, sometimes hurting performance for one demographic while helping performance for others. The inequitable benefits of in-context learning for SRM present a challenge for practitioners implementing SRMs, and for decision-makers who might come to rely on them. Our work highlights a need for fine-grained benchmarks captured from diverse subpopulations that test not only fidelity but generalization.Keywords
All Related Versions
This publication has 11 references indexed in Scilit:
- A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and OpportunitiesACM Computing Surveys, 2023
- Out of One, Many: Using Language Models to Simulate Human SamplesPolitical Analysis, 2023
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language ProcessingACM Computing Surveys, 2023
- Improving Diversity of Demographic Representation in Large Language Models via Collective-Critiques and Self-Voting2023
- WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipedia2023
- Large language models are few-shot clinical information extractors2022
- Best practices in collecting online data with Asian, Black, Latino, and White respondents: evidence from the 2016 Collaborative Multiracial Post-election SurveyPolitics, Groups, and Identities, 2017
- Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction InstrumentsBig Data, 2017
- Public Opinion and American DemocracyPublic Opinion Quarterly, 2011
- The use of MMR, diversity-based reranking for reordering documents and producing summaries1998