Klint Kanopka
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Klint Kanopka has a total of 85 co-authors. Here are the top co-authors by number of shared publications:
- Benjamin W. Domingue
Graduate School of Education, Stanford University, Stanford, CA, USA
20 shared publications - Sam Trejo
Office of Population Research and Department of Sociology, Princeton University, Princeton, NJ, USA
6 shared publications - Lijin Zhang
Graduate School of Education, Stanford University, Stanford, CA, USA
5 shared publications Radhika Kapoor
Graduate School of Education, Stanford University, Santa Clara, USA
5 shared publications- Charles Rahal
Leverhulme Centre for Demographic Science and Nuffield College, University of Oxford, Oxford, UK
4 shared publications - Elliot M. Tucker-Drob
Department of Psychology, The University of Texas at Austin
4 shared publications
Latest Publications
- Open Access
- 6 November 2026
- Published by Center for Open Science in PsyArXiv
What Is a Wrong Answer Worth? The Out-of-Sample Value of the Chosen Option in Multiple-Choice Tests
Right/wrong scoring of multiple-choice items discards information about which wrong option was chosen. IRT models that use the choice date from 1972, but evidence on its value has come mostly in sample, one test at a time. We compared option-level and right/wrong models on predicting the option chosen in held-out responses (log-loss, in bits) across 116 Item Response Warehouse tables (110 Brazilian), 469 TIMSS and PIRLS tests and 121 Czech exam forms. In the warehouse, modeling the chosen option gained 15.4 millibits per response over the 2PL, 1.36 times the gain from estimating item discrimination. Of the option's gain, 84% came from which wrong option was chosen, where right/wrong models use only distractor popularity; the rest, on whether the answer was right, was about a fifth of discrimination's gain. The gain was positive in all 116 warehouse tables, in 119 of 121 Czech forms and, with priors on both models, in 463 of 469 TIMSS and PIRLS tests; relative to discrimination, it ranged from 0.8 (Czech exams) to 1.5 (ENEM). By our declared measure, the gain peaked in the lowest ability deciles in most tables; per wrong answer, the choice improved prediction over distractor popularity most in the top decile.
- 6 November 2026
- Published by Center for Open Science in PsyArXiv
Open AccessWhat Is a Wrong Answer Worth? The Out-of-Sample Value of the Chosen Option in Multiple-Choice Tests
Right/wrong scoring of multiple-choice items discards information about which wrong option was chosen. IRT models that use the choice date from 1972, but evidence on its value has come mostly in sample, one test at a time. We compared option-level and right/wrong models on predicting the option chosen in held-out responses (log-loss, in bits) across 116 Item Response Warehouse tables (110 Brazilian), 469 TIMSS and PIRLS tests and 121 Czech exam forms. In the warehouse, modeling the chosen option gained 15.4 millibits per response over the 2PL, 1.36 times the gain from estimating item discrimination. Of the option's gain, 84% came from which wrong option was chosen, where right/wrong models use only distractor popularity; the rest, on whether the answer was right, was about a fifth of discrimination's gain. The gain was positive in all 116 warehouse tables, in 119 of 121 Czech forms and, with priors on both models, in 463 of 469 TIMSS and PIRLS tests; relative to discrimination, it ranged from 0.8 (Czech exams) to 1.5 (ENEM). By our declared measure, the gain peaked in the lowest ability deciles in most tables; per wrong answer, the choice improved prediction over distractor popularity most in the top decile.
- 6 July 2026
- Published by The R Foundation
meow: Unified Framework for Computer Adaptive Testing Simulations
Provides an extensible framework for conducting simulations to compare data generating processes, item selection algorithms, parameter update algorithms, and stopping rules in computer adaptive testing (CAT) applications. Bundled algorithms include the Elo-based update rules of Klinkenberg, Straatemeier and van der Maas (2011) and Vermeiren, Kruis, Bolsinova, van der Maas and Hofman (2025) .
- 6 July 2026
- Published by The R Foundation
meow: Unified Framework for Computer Adaptive Testing Simulations
Provides an extensible framework for conducting simulations to compare data generating processes, item selection algorithms, parameter update algorithms, and stopping rules in computer adaptive testing (CAT) applications. Bundled algorithms include the Elo-based update rules of Klinkenberg, Straatemeier and van der Maas (2011) and Vermeiren, Kruis, Bolsinova, van der Maas and Hofman (2025) .
- 25 June 2026
- Published by The R Foundation
mixedsubjectsirt: Item Response Theory Calibration with a Mixed Subjects Design
- 25 June 2026
- Published by The R Foundation
mixedsubjectsirt: Item Response Theory Calibration with a Mixed Subjects Design
- Open Access
- 17 June 2026
- Published by Center for Open Science in PsyArXiv
Making Room for Theory in DIF and Measurement Invariance Analysis with Covariate Effects on Groups of Items
We propose and study an extension of moderated nonlinear factor analysis that models covariate effects—that is, differential item functioning (DIF)—at the level of item groups rather than, or in addition to, individual items. This allows for a more theory-driven approach to DIF and measurement invariance testing in which researchers can form and test hypotheses about why items differ in how they function across an arbitrary number of categorical or continuous covariates. As a motivating example, we demonstrate gender effects on a developmental measure where scores that assume invariance overstate boys’ cognitive development and understate their fine and gross motor development, with opposite implications for girls. Simulation demonstrates the empirical identifiability and predictable statistical performance of the proposed model, and additional empirical examples demonstrate the approach’s substantial flexibility.
- 17 June 2026
- Published by Center for Open Science in PsyArXiv
Open AccessMaking Room for Theory in DIF and Measurement Invariance Analysis with Covariate Effects on Groups of Items
We propose and study an extension of moderated nonlinear factor analysis that models covariate effects—that is, differential item functioning (DIF)—at the level of item groups rather than, or in addition to, individual items. This allows for a more theory-driven approach to DIF and measurement invariance testing in which researchers can form and test hypotheses about why items differ in how they function across an arbitrary number of categorical or continuous covariates. As a motivating example, we demonstrate gender effects on a developmental measure where scores that assume invariance overstate boys’ cognitive development and understate their fine and gross motor development, with opposite implications for girls. Simulation demonstrates the empirical identifiability and predictable statistical performance of the proposed model, and additional empirical examples demonstrate the approach’s substantial flexibility.
- Open Access
- 6 June 2026
- Published by Center for Open Science in SocArXiv
Comparing Campaign Rhetoric and Legislative Behavior Through Issue-Specific Polarization in the US House of Representatives, 2021–2025
Voters make decisions on who to elect based, in part, upon candidate messaging and party platforms. Here we examine the relationship between campaign rhetoric and legislative behavior among members of the 117th and 118th United States House of Representatives. Using campaign text and roll-call voting data, the paper compares polarization across policy domains through an IssueIRT model of legislative behavior and a Wordscores model of campaign rhetoric. We find that polarization varies substantially across issue areas and that the relationship between rhetoric and behavior is not uniform. Civil Rights exhibits highly polarized and morally distinct partisan rhetoric, while Government Operations rhetoric displays a more continuous ideological spectrum despite strongly polarized voting behavior. International Affairs appears comparatively less polarized in both rhetoric and legislative behavior, with greater thematic overlap between parties surrounding national security and international engagement. We also observe a second latent dimension in congressional voting associated with institutionalism and intra-party factionalism. Republican legislators exhibit substantially greater dispersion along this dimension, while Democrats remain comparatively cohesive in both rhetoric and voting behavior.
- 6 June 2026
- Published by Center for Open Science in SocArXiv
Open AccessComparing Campaign Rhetoric and Legislative Behavior Through Issue-Specific Polarization in the US House of Representatives, 2021–2025
Voters make decisions on who to elect based, in part, upon candidate messaging and party platforms. Here we examine the relationship between campaign rhetoric and legislative behavior among members of the 117th and 118th United States House of Representatives. Using campaign text and roll-call voting data, the paper compares polarization across policy domains through an IssueIRT model of legislative behavior and a Wordscores model of campaign rhetoric. We find that polarization varies substantially across issue areas and that the relationship between rhetoric and behavior is not uniform. Civil Rights exhibits highly polarized and morally distinct partisan rhetoric, while Government Operations rhetoric displays a more continuous ideological spectrum despite strongly polarized voting behavior. International Affairs appears comparatively less polarized in both rhetoric and legislative behavior, with greater thematic overlap between parties surrounding national security and international engagement. We also observe a second latent dimension in congressional voting associated with institutionalism and intra-party factionalism. Republican legislators exhibit substantially greater dispersion along this dimension, while Democrats remain comparatively cohesive in both rhetoric and voting behavior.
- Open Access
- 30 April 2026
- Published by Center for Open Science in PsyArXiv
A Markov Chain Monte Carlo Procedure with Simulated Annealing for Shortening Assessments
Symptom screeners, patient assessments, and cognitive tests are used to measure constructs such as health-related quality of life, levels of mental health symptoms, and ability. The items are often summarized, through a simple unweighted sum score or through more advanced methods, including factor analysis, Rasch modeling, and item response theory. However, long instruments increase respondent burden and it may be desirable to shorten them. One consideration that is often overlooked is that in many contexts, these instruments are used less for precise measurement of a latent construct but instead for accurate prediction of a diagnosis, expert judgment, or classification. As such, we present a method to shorten instruments that prioritizes reproduction of expert judgment based on an unweighted sum score through the use of a Markov Chain Monte Carlo algorithm with simulated annealing to explore the search space of potential short forms. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be perfectly aligned with the decision criteria of interest. We prove the correctness of the algorithm, demonstrate its effectiveness and efficiency through simulation studies, and apply it to a screener for alcohol use disorder, demonstrating that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.
- 30 April 2026
- Published by Center for Open Science in PsyArXiv
Open AccessA Markov Chain Monte Carlo Procedure with Simulated Annealing for Shortening Assessments
Symptom screeners, patient assessments, and cognitive tests are used to measure constructs such as health-related quality of life, levels of mental health symptoms, and ability. The items are often summarized, through a simple unweighted sum score or through more advanced methods, including factor analysis, Rasch modeling, and item response theory. However, long instruments increase respondent burden and it may be desirable to shorten them. One consideration that is often overlooked is that in many contexts, these instruments are used less for precise measurement of a latent construct but instead for accurate prediction of a diagnosis, expert judgment, or classification. As such, we present a method to shorten instruments that prioritizes reproduction of expert judgment based on an unweighted sum score through the use of a Markov Chain Monte Carlo algorithm with simulated annealing to explore the search space of potential short forms. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be perfectly aligned with the decision criteria of interest. We prove the correctness of the algorithm, demonstrate its effectiveness and efficiency through simulation studies, and apply it to a screener for alcohol use disorder, demonstrating that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.
- Open Access
- 15 April 2026
- Published by Center for Open Science in PsyArXiv
Understanding the evolution of individual response process: An exploratory approach
The massive use of computer-adaptive testing has contributed to revolutionizing psychometric analysis in terms of both data collection and analysis methodologies. The National Assessment of Educational Progress (NAEP) is one of the most well-known assessments that collect respondents' process data, which has been extensively analyzed (Bosch, 2021; Levin, 2021; Patel et al., 2021; Zehner et al., 2020). Much of this work has primarily focused on efficiency estimation, due to the 2019 NAEP Educational Data Mining Competition. Prediction models were trained mainly using item-level feature engineering and person-level grouping. Some work focusing on exploring the data itself rather than training models has also focused on grouping respondents' behavior patterns on subsets of items (Wei et al., 2024) through the use of clustering and relating these patterns to demographics and respondent ability. With a similar goal, we expand on this approach by firstly developing item level behavior profiles using clustering-based methods (i.e., latent class analysis) for all items. Using these profiles, we explore individual trajectories from profile to profile over the course of the test. Combining these trajectories with latent abilities and speed estimated from item responses and demographic information, we characterize the respondents who comprise the individual trajectories. Preliminary results find item-to-item variation in the structure and interpretation of the behavior profiles, but also distinct profiles within each item that represent differences in the underlying IRT-derived latent ability distributions. Understanding usage patterns for respondents of various ability levels, demographic backgrounds, and disability statuses may inform future item development, assistive tool design, and user interface for computerized assessments.
- 15 April 2026
- Published by Center for Open Science in PsyArXiv
Open AccessUnderstanding the evolution of individual response process: An exploratory approach
The massive use of computer-adaptive testing has contributed to revolutionizing psychometric analysis in terms of both data collection and analysis methodologies. The National Assessment of Educational Progress (NAEP) is one of the most well-known assessments that collect respondents' process data, which has been extensively analyzed (Bosch, 2021; Levin, 2021; Patel et al., 2021; Zehner et al., 2020). Much of this work has primarily focused on efficiency estimation, due to the 2019 NAEP Educational Data Mining Competition. Prediction models were trained mainly using item-level feature engineering and person-level grouping. Some work focusing on exploring the data itself rather than training models has also focused on grouping respondents' behavior patterns on subsets of items (Wei et al., 2024) through the use of clustering and relating these patterns to demographics and respondent ability. With a similar goal, we expand on this approach by firstly developing item level behavior profiles using clustering-based methods (i.e., latent class analysis) for all items. Using these profiles, we explore individual trajectories from profile to profile over the course of the test. Combining these trajectories with latent abilities and speed estimated from item responses and demographic information, we characterize the respondents who comprise the individual trajectories. Preliminary results find item-to-item variation in the structure and interpretation of the behavior profiles, but also distinct profiles within each item that represent differences in the underlying IRT-derived latent ability distributions. Understanding usage patterns for respondents of various ability levels, demographic backgrounds, and disability statuses may inform future item development, assistive tool design, and user interface for computerized assessments.
- Open Access
- 3 April 2026
- Published by Center for Open Science in SocArXiv
Using large language models as a source of human behavioral data in social science experiments
Large language models (LLMs) have prompted proposals to replace human subjects in social science experiments with simulated responses. Empirical evaluations suggest that this practice---often called silicon sampling---can sometimes approximate human behavior but is unreliable. We delineate where this approach may still provide value and where it may not, but primarily study an alternative approach: one in which model-based predictions are used not as substitutes for human data, but as auxiliary measurements within randomized experiments. We formalize the inference of causal estimands from mixed-subjects randomized controlled trials, in which outcomes are observed for a subset of units while predictions are available for all units. Under transparent design conditions, we derive a family of estimators that remain unbiased for the average treatment effect in finite samples while exploiting predictions to reduce variance. We characterize when prediction-powered, calibration-based, arm-specifically tuned, and difference-in-predictions estimators improve precision, and we provide a software package which operationalizes these results and aids researchers to jointly select estimators and allocate budgets between human data collection and prediction generation. Together, our results show how generative artificial intelligence can improve experimental social science without compromising scientific validity.
- 3 April 2026
- Published by Center for Open Science in SocArXiv
Open AccessUsing large language models as a source of human behavioral data in social science experiments
Large language models (LLMs) have prompted proposals to replace human subjects in social science experiments with simulated responses. Empirical evaluations suggest that this practice---often called silicon sampling---can sometimes approximate human behavior but is unreliable. We delineate where this approach may still provide value and where it may not, but primarily study an alternative approach: one in which model-based predictions are used not as substitutes for human data, but as auxiliary measurements within randomized experiments. We formalize the inference of causal estimands from mixed-subjects randomized controlled trials, in which outcomes are observed for a subset of units while predictions are available for all units. Under transparent design conditions, we derive a family of estimators that remain unbiased for the average treatment effect in finite samples while exploiting predictions to reduce variance. We characterize when prediction-powered, calibration-based, arm-specifically tuned, and difference-in-predictions estimators improve precision, and we provide a software package which operationalizes these results and aids researchers to jointly select estimators and allocate budgets between human data collection and prediction generation. Together, our results show how generative artificial intelligence can improve experimental social science without compromising scientific validity.
- 30 March 2026
- Published by Taylor & Francis in Multivariate Behavioral Research
- Vol. 61 (5) , 1-20
Evaluating Model Predictive Performance in Confirmatory Factor Analysis with Binary Outcomes Using the InterModel Vigorish
In Structural Equation Modeling (SEM), it is a common practice to compare models so as to identify the one that best fits the data. Various indices for assessing model fit have been proposed to aid...
- 30 March 2026
- Published by Taylor & Francis in Multivariate Behavioral Research
- Vol. 61 (5) , 1-20
Evaluating Model Predictive Performance in Confirmatory Factor Analysis with Binary Outcomes Using the InterModel Vigorish
In Structural Equation Modeling (SEM), it is a common practice to compare models so as to identify the one that best fits the data. Various indices for assessing model fit have been proposed to aid...
- Open Access
- 15 December 2025
- Published by Center for Open Science in PsyArXiv
Realistic Simulation of Item Difficulties
Simulation studies are commonly used to improve understanding of psychometric models. For many common models, an essential feature of the simulation is the relative (to variation in person parameters) variation in item difficulties. A common practice has been to generate both item difficulty parameters and person parameters directly from a standard normal distribution or to otherwise stipulate distributions for both parameters at the outset of the simulation. This approach implicitly assumes that these distributions can adequately represent empirical data—an assumption that warrants careful examination. In this paper, leveraging 73 datasets from the Item Response Warehouse (Domingue et al., 2023), we examine the variability of item difficulty distributions in real-world datasets and investigate how this variation influences estimation and simulation. We identify key distributional characteristics (e.g., variance and skewness) and propose a new method for simulating realistic item difficulties based on empirical data. This method enhances the realism and applicability of simulation results, making them more reflective of real-world measurement conditions and improving the robustness of psychometric model evaluation
- 15 December 2025
- Published by Center for Open Science in PsyArXiv
Open AccessRealistic Simulation of Item Difficulties
Simulation studies are commonly used to improve understanding of psychometric models. For many common models, an essential feature of the simulation is the relative (to variation in person parameters) variation in item difficulties. A common practice has been to generate both item difficulty parameters and person parameters directly from a standard normal distribution or to otherwise stipulate distributions for both parameters at the outset of the simulation. This approach implicitly assumes that these distributions can adequately represent empirical data—an assumption that warrants careful examination. In this paper, leveraging 73 datasets from the Item Response Warehouse (Domingue et al., 2023), we examine the variability of item difficulty distributions in real-world datasets and investigate how this variation influences estimation and simulation. We identify key distributional characteristics (e.g., variance and skewness) and propose a new method for simulating realistic item difficulties based on empirical data. This method enhances the realism and applicability of simulation results, making them more reflective of real-world measurement conditions and improving the robustness of psychometric model evaluation