Evaluating the construct validity of text embeddings with application to survey questions
Authors:
Open Access
- 7 July 2022
- journal article
- research article
- Published by Springer Nature in EPJ Data Science
- Vol. 11 (1) , 1-31
Abstract
Text embedding models from Natural Language Processing can map text data (e.g. words, sentences, documents) to meaningful numerical representations (a.k.a. text embeddings). While such models are increasingly applied in social science research, one important issue is often not addressed: the extent to which these embeddings are high-quality representations of the information needed to be encoded. We view this quality evaluation problem from a measurement validity perspective, and propose the use of the classic construct validity framework to evaluate the quality of text embeddings. First, we describe how this framework can be adapted to the opaque and high-dimensional nature of text embeddings. Second, we apply our adapted framework to an example where we compare the validity of survey question representation across text embedding models.Keywords
All Related Versions
Funding Information
- Nederlandse Organisatie voor Wetenschappelijk Onderzoek (VI.Vidi.195.152, VI.Veni.192.130)
This publication has 28 references indexed in Scilit:
- Word embeddings quantify 100 years of gender and ethnic stereotypesProceedings of the National Academy of Sciences, 2018
- Semantics derived automatically from language corpora contain human-like biasesScience, 2017
- Issues in evaluating semantic spaces using word analogies2016
- Defining Words with Words: Beyond the Distributional Hypothesis2016
- The Elements of Statistical Learning2009
- Fast times and easy questions: the effects of age, experience and question complexity on web survey response timesApplied Cognitive Psychology, 2007
- Design, Evaluation, and Analysis of Questionnaires for Survey Research2007
- Are Surveys on Trust Trustworthy?Social Psychology Quarterly, 2003
- Random ForestsMachine Learning, 2001
- Distributional StructureWORD, 1954