KKKlint Kanopka

Klint Kanopka

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Klint Kanopka has a total of 85 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    What Is a Wrong Answer Worth? The Out-of-Sample Value of the Chosen Option in Multiple-Choice Tests

    by Santiago Rivadeneira Quintero, Klint Kanopka, Mateus Mazzaferro, Benjamin Domingue

    Right/wrong scoring of multiple-choice items discards information about which wrong option was chosen. IRT models that use the choice date from 1972, but evidence on its value has come mostly in sample, one test at a time. We compared option-level and right/wrong models on predicting the option chosen in held-out responses (log-loss, in bits) across 116 Item Response Warehouse tables (110 Brazilian), 469 TIMSS and PIRLS tests and 121 Czech exam forms. In the warehouse, modeling the chosen option gained 15.4 millibits per response over the 2PL, 1.36 times the gain from estimating item discrimination. Of the option's gain, 84% came from which wrong option was chosen, where right/wrong models use only distractor popularity; the rest, on whether the answer was right, was about a fifth of discrimination's gain. The gain was positive in all 116 warehouse tables, in 119 of 121 Czech forms and, with priors on both models, in 463 of 469 TIMSS and PIRLS tests; relative to discrimination, it ranged from 0.8 (Czech exams) to 1.5 (ENEM). By our declared measure, the gain peaked in the lowest ability deciles in most tables; per wrong answer, the choice improved prediction over distractor popularity most in the top decile.

  • meow: Unified Framework for Computer Adaptive Testing Simulations

    by Klint Kanopka, Sophia Deng

    Provides an extensible framework for conducting simulations to compare data generating processes, item selection algorithms, parameter update algorithms, and stopping rules in computer adaptive testing (CAT) applications. Bundled algorithms include the Elo-based update rules of Klinkenberg, Straatemeier and van der Maas (2011) and Vermeiren, Kruis, Bolsinova, van der Maas and Hofman (2025) .

  • Open Access

    Making Room for Theory in DIF and Measurement Invariance Analysis with Covariate Effects on Groups of Items

    by Sanford R Student, Klint Kanopka

    We propose and study an extension of moderated nonlinear factor analysis that models covariate effects—that is, differential item functioning (DIF)—at the level of item groups rather than, or in addition to, individual items. This allows for a more theory-driven approach to DIF and measurement invariance testing in which researchers can form and test hypotheses about why items differ in how they function across an arbitrary number of categorical or continuous covariates. As a motivating example, we demonstrate gender effects on a developmental measure where scores that assume invariance overstate boys’ cognitive development and understate their fine and gross motor development, with opposite implications for girls. Simulation demonstrates the empirical identifiability and predictable statistical performance of the proposed model, and additional empirical examples demonstrate the approach’s substantial flexibility.

  • Open Access

    Comparing Campaign Rhetoric and Legislative Behavior Through Issue-Specific Polarization in the US House of Representatives, 2021–2025

    by Jillian Pizza, Klint Kanopka

    Voters make decisions on who to elect based, in part, upon candidate messaging and party platforms. Here we examine the relationship between campaign rhetoric and legislative behavior among members of the 117th and 118th United States House of Representatives. Using campaign text and roll-call voting data, the paper compares polarization across policy domains through an IssueIRT model of legislative behavior and a Wordscores model of campaign rhetoric. We find that polarization varies substantially across issue areas and that the relationship between rhetoric and behavior is not uniform. Civil Rights exhibits highly polarized and morally distinct partisan rhetoric, while Government Operations rhetoric displays a more continuous ideological spectrum despite strongly polarized voting behavior. International Affairs appears comparatively less polarized in both rhetoric and legislative behavior, with greater thematic overlap between parties surrounding national security and international engagement. We also observe a second latent dimension in congressional voting associated with institutionalism and intra-party factionalism. Republican legislators exhibit substantially greater dispersion along this dimension, while Democrats remain comparatively cohesive in both rhetoric and voting behavior.

  • Open Access

    A Markov Chain Monte Carlo Procedure with Simulated Annealing for Shortening Assessments

    by Klint Kanopka, Xiaoxuan Zhang, Daphna Harel

    Symptom screeners, patient assessments, and cognitive tests are used to measure constructs such as health-related quality of life, levels of mental health symptoms, and ability. The items are often summarized, through a simple unweighted sum score or through more advanced methods, including factor analysis, Rasch modeling, and item response theory. However, long instruments increase respondent burden and it may be desirable to shorten them. One consideration that is often overlooked is that in many contexts, these instruments are used less for precise measurement of a latent construct but instead for accurate prediction of a diagnosis, expert judgment, or classification. As such, we present a method to shorten instruments that prioritizes reproduction of expert judgment based on an unweighted sum score through the use of a Markov Chain Monte Carlo algorithm with simulated annealing to explore the search space of potential short forms. This method is orders of magnitude more efficient than brute force search and has the advantage of optimizing for prediction of the desired outcome directly, instead of optimizing on the measurement of an intermediate outcome that may not be perfectly aligned with the decision criteria of interest. We prove the correctness of the algorithm, demonstrate its effectiveness and efficiency through simulation studies, and apply it to a screener for alcohol use disorder, demonstrating that under the unweighted sum score scoring constraint, shortened forms can more accurately predict expert judgments than full forms.

  • Open Access

    Understanding the evolution of individual response process: An exploratory approach

    by Ruiting Shen, Klint Kanopka

    The massive use of computer-adaptive testing has contributed to revolutionizing psychometric analysis in terms of both data collection and analysis methodologies. The National Assessment of Educational Progress (NAEP) is one of the most well-known assessments that collect respondents' process data, which has been extensively analyzed (Bosch, 2021; Levin, 2021; Patel et al., 2021; Zehner et al., 2020). Much of this work has primarily focused on efficiency estimation, due to the 2019 NAEP Educational Data Mining Competition. Prediction models were trained mainly using item-level feature engineering and person-level grouping. Some work focusing on exploring the data itself rather than training models has also focused on grouping respondents' behavior patterns on subsets of items (Wei et al., 2024) through the use of clustering and relating these patterns to demographics and respondent ability. With a similar goal, we expand on this approach by firstly developing item level behavior profiles using clustering-based methods (i.e., latent class analysis) for all items. Using these profiles, we explore individual trajectories from profile to profile over the course of the test. Combining these trajectories with latent abilities and speed estimated from item responses and demographic information, we characterize the respondents who comprise the individual trajectories. Preliminary results find item-to-item variation in the structure and interpretation of the behavior profiles, but also distinct profiles within each item that represent differences in the underlying IRT-derived latent ability distributions. Understanding usage patterns for respondents of various ability levels, demographic backgrounds, and disability statuses may inform future item development, assistive tool design, and user interface for computerized assessments.

  • Open Access

    Using large language models as a source of human behavioral data in social science experiments

    by Austin van Loon, Klint Kanopka

    Large language models (LLMs) have prompted proposals to replace human subjects in social science experiments with simulated responses. Empirical evaluations suggest that this practice---often called silicon sampling---can sometimes approximate human behavior but is unreliable. We delineate where this approach may still provide value and where it may not, but primarily study an alternative approach: one in which model-based predictions are used not as substitutes for human data, but as auxiliary measurements within randomized experiments. We formalize the inference of causal estimands from mixed-subjects randomized controlled trials, in which outcomes are observed for a subset of units while predictions are available for all units. Under transparent design conditions, we derive a family of estimators that remain unbiased for the average treatment effect in finite samples while exploiting predictions to reduce variance. We characterize when prediction-powered, calibration-based, arm-specifically tuned, and difference-in-predictions estimators improve precision, and we provide a software package which operationalizes these results and aids researchers to jointly select estimators and allocate budgets between human data collection and prediction generation. Together, our results show how generative artificial intelligence can improve experimental social science without compromising scientific validity.

  • Evaluating Model Predictive Performance in Confirmatory Factor Analysis with Binary Outcomes Using the InterModel Vigorish

    by Lijin Zhang, Charles Rahal, Klint Kanopka, Esther Ulitzsch, Zhiyong Zhang, Benjamin W. Domingue

    In Structural Equation Modeling (SEM), it is a common practice to compare models so as to identify the one that best fits the data. Various indices for assessing model fit have been proposed to aid...


  • Open Access

    Realistic Simulation of Item Difficulties

    by Lijin Zhang, Yiqing Liu, Dylan Molenaar, Joshua Gilbert, Klint Kanopka, Benjamin W. Domingue

    Simulation studies are commonly used to improve understanding of psychometric models. For many common models, an essential feature of the simulation is the relative (to variation in person parameters) variation in item difficulties. A common practice has been to generate both item difficulty parameters and person parameters directly from a standard normal distribution or to otherwise stipulate distributions for both parameters at the outset of the simulation. This approach implicitly assumes that these distributions can adequately represent empirical data—an assumption that warrants careful examination. In this paper, leveraging 73 datasets from the Item Response Warehouse (Domingue et al., 2023), we examine the variability of item difficulty distributions in real-world datasets and investigate how this variation influences estimation and simulation. We identify key distributional characteristics (e.g., variance and skewness) and propose a new method for simulating realistic item difficulties based on empirical data. This method enhances the realism and applicability of simulation results, making them more reflective of real-world measurement conditions and improving the robustness of psychometric model evaluation