SMSterett H. Mercer

Sterett H. Mercer

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Sterett H. Mercer has a total of 131 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Adaptive Pair Selection in Large Language Model–Based Comparative Judgment of Writing

    by Sterett H. Mercer

    Comparative judgment assesses a set of writing responses by repeatedly asking which of two is stronger on a specified trait, then combining judgments to estimate their relative quality. Using 676,326 large language model (LLM) judgments of 1,714 essays across 16 prompts and six traits, this study compared four adaptive pairing strategies (random, maximum ambiguity, moderate ambiguity, and hybrid) at 2–16 comparisons per essay. Two Bayesian and three frequentist Bradley–Terry estimators were compared on agreement with scores based on all comparisons and prediction of withheld judgments; stopping rules were assessed for the Bayesian and conventional frequentist estimators. Adaptive pairing improved recovery efficiency, particularly at intermediate comparison counts, but hybrid pairing offered no clear advantage over simpler strategies. Neither Bayesian estimator reached .90 reliability at the highest comparison count, although 74%–89% of comparison sequences achieved squared correlations of at least .90 with the full-comparison scores. Conventional frequentist reliability estimates were unstable at low comparison counts and sometimes indicated stopping too early. No estimator performed best across recovery, prediction, and score spread. Effective use of LLM-based adaptive comparative judgment therefore requires coordinated choices about pair selection strategy, score estimation, and stopping criteria.

  • Open Access

    Screening elementary writing with AI: Evidence for ChatGPT as a feasible alternative to human ratings and specialized scoring models

    by Michael Matta, Milena A. Keller-Margulis, Sterett H. Mercer

    Universal writing screening requires assessments that are both valid and feasible for school use. This study evaluated whether general-purpose artificial intelligence (AI) could automate comparative judgment (CJ) of student writing for screening. Specifically, the study examined agreement between ChatGPT-generated and human CJ scores, the extent to which ChatGPT-generated CJ scores (ChatGPT-CJ) predicted performance on standardized writing assessments, and whether these relations differed between emerging bilingual (EB) and non-EB students. ChatGPT-CJ scores were also compared with feature-based automated scoring systems and curriculum-based writing measures. Participants were 237 students in Grades 3-5 (n = 134 EB students, 57%) from five elementary schools in the southwestern United States. Students completed three writing screening tasks scored using human CJ, ChatGPT-CJ, writeAlizer, Project Essay Grade (PEG), and Correct Minus Incorrect Word Sequences (CIWS). Standardized writing assessments served as criterion measures. Agreement between human and ChatGPT judgments of identical writing pairs ranged from 76% to 81% (κ = .51-.61). ChatGPT-CJ scores were strongly associated with human CJ scores (r = .91-.92) and criterion writing quality (β = .96-.97). Relative to other scoring methods, ChatGPT-CJ scores demonstrated performance comparable to or stronger than human CJs, writeAlizer, PEG, and CIWS. No statistically significant differences in criterion validity were observed between EB and non-EB students. No practical differences were observed between ChatGPT models with reasoning enabled vs. disabled. Findings suggest general-purpose AI can generate writing quality scores that closely align with human judgments and provide valid indicators for universal screening.

  • Open Access

    Identifying optimal indicators of performance using automated written-expression curriculum-based measurement: An examination of genre effects

    by Milena Keller-Margulis, Sterett Mercer, Michael Matta, Brenda Duran

    Efficient, technically adequate screeners are needed to identify students at risk for writing difficulties, yet traditional written-expression curriculum-based measurement indicators and rubric-based approaches often present feasibility and validity challenges. This study examined the validity of automated writing quality scores generated with writeAlizer, an open-source automated tool, across 15-min narrative, expository, and persuasive samples from 244 third- through fifth-grade students. Human ratings of writing quality were generated using paired comparisons, and automated scores were produced using genre-specific and -nonspecific writeAlizer models. Validity evidence was evaluated through correlations with human ratings and norm-referenced criterion measures, in comparison to a commercial automated tool, Project Essay Grade (PEG). Results indicated that writeAlizer scores strongly correlated (r > .90) with human ratings, with no clear advantage for genre-specific scoring models. Scores from individual screening samples demonstrated moderate relations with criterion measures (r = .40 to .68) and performed similarly to PEG. Screening validity was strongest when multiple WE-CBM samples were modeled as indicators of latent screening quality predicting latent criterion quality (r = .88). Findings support automated scoring as a promising approach to improve the feasibility of writing screening and suggest that multiple writing samples may be needed for technically adequate decision-making.

  • Open Access

    Comparative Judgment Versus Direct Rubric Scoring With Large Language Models: Effects of Human-Scored Reference Samples

    by Sterett Mercer

    Human-scored writing samples can support large language model (LLM) scoring as exemplars for direct rubric scoring or as references for calibrating comparative judgment (CJ). This study compared these uses with the same reference essays at each matched reference set size. GPT-5.6 Terra scored 1,714 essays from students in Grades 8-12 across 16 ELLIPSE prompts on six analytic writing traits. Direct scoring used 0-80 references; CJ used Bayesian Bradley-Terry-Luce scaling with cumulative-logit calibration at 20, 40, and 80 references. Quadratic weighted kappa (QWK) for direct scoring increased from .30 without references to .55 with 80, whereas CJ QWK was already .55 at 20 references and changed little thereafter; its initial advantage was essentially eliminated by 80 references. At 80 references, CJ showed a higher Pearson correlation with human scores (.62 vs. .59), whereas direct scoring showed lower mean absolute error (.47 vs. .57) and higher within-one-point agreement (.96 vs. .91). Performance differed across traits, with the clearest CJ advantage for Phraseology, and no demographic moderation of relative absolute error was detected. Across metrics, CJ better preserved relative differences among essays, whereas direct scoring more closely reproduced human-assigned rubric scores.

  • Open Access

    Validity of Large Language Model Comparative Judgment for Universal Writing Screening

    by Sterett Mercer, Deborah K. Reed

    Universal writing screening requires scoring approaches that are feasible and supported by validity evidence. This study evaluated large language model (LLM)-based comparative judgment (CJ) for scoring informational writing assessments completed by 1,208 students in Grades 3–6 across three screening occasions. Seven LLMs representing different capability and cost tiers performed pairwise comparisons of writing quality. LLM-based CJ scores showed meaningful convergence with researcher analytic rubric scores (r = .59–.73). For single-wave scores, LLM-based CJ was comparable to researcher scoring in predicting state writing rubric scores and generally matched or exceeded it in predicting ELA scale scores and classifying ELA proficiency. Averaging scores across three screening waves substantially improved criterion-related validity and classification accuracy, with LLM-based CJ reaching β = .59–.66 for state writing rubric scores, β = .68–.74 for ELA scale scores, and AUC = .82–.86 for ELA proficiency. Predictive bias patterns for multilingual learners were similar across scoring methods. Findings were broadly consistent across LLMs, with little evidence that greater model capability or cost improved validity evidence. Results support LLM-based CJ as a promising approach for efficient writing screening and highlight the value of multiple writing samples per student and task-specific evaluation of validity relative to cost.

  • pairwiseLLM: Pairwise Comparison Tools for Large Language Model-Based Writing Evaluation

    by Sterett H. Mercer

    Provides a unified framework for generating, submitting, and analyzing pairwise comparisons of writing quality using large language models (LLMs). The package supports live and/or batch evaluation workflows across multiple providers ('OpenAI', 'Anthropic', 'Google Gemini', 'Together AI', and locally-hosted 'Ollama' models), includes bias-tested prompt templates and a flexible template registry, and offers tools for constructing forward and reversed comparison sets to analyze consistency and positional bias. Results can be modeled using Bradley–Terry (1952) or Elo rating methods to derive writing quality scores. For information on the method of pairwise comparisons, see Thurstone (1927) and Heldsinger & Humphry (2010) . For information on Elo ratings, see Clark et al. (2018) .


  • Improving written-expression curriculum-based measurement feasibility with automated writing evaluation programs.

    by Michael Matta, Milena A. Keller-Margulis, Sterett H. Mercer

    Automated writing evaluation programs have emerged as alternative, feasible approaches for scoring student writing. This study evaluated accuracy, predictive validity, diagnostic accuracy, and bias of automated scores of Written-Expression Curriculum-Based Measurement (WE-CBM). A sample of 722 students in Grades 2-5 completed 3-min WE-CBM tasks during one school year. A subset of students also completed the state-mandated writing test the same year or 1 year later. Writing samples were hand-scored for four WE-CBM metrics. A computer-based approach generated automated scores for the same four metrics. Findings indicate simpler automated metrics such as total words written and words spelled correctly, closely matched hand-calculated scores, while small differences were observed for more complex metrics including correct word sequences and correct minus incorrect word sequences. Automated scores for simpler WE-CBM metrics also predicted performance on the state test similarly to hand-calculated scores. Finally, we failed to identify evidence of bias between African American and Hispanic students associated with automated scores. Implications of using automated scores for educational decision making are discussed. (PsycInfo Database Record (c) 2025 APA, all rights reserved).


  • Predictors of Sustained Implementation of a Universal Behavior Support Framework Through the COVID-19 Pandemic

    by Kent McIntosh, Angus Kittelman, Sterett H. Mercer, María Reina Santiago-Rosario, Sean C. Austin, Robert H. Horner, Heather Peshak George

    In this 5-year prospective study, we identified practice, school, and district variables predicting sustained implementation of positive behavioral interventions and supports (PBIS), a widely implemented framework for improving social, emotional, behavioral, and academic outcomes. Uniquely, the study captured implementation before and after the onset of the COVID-19 pandemic. Personnel from 595 schools across 22 U.S. states completed a validated measure of implementation strategies during the 2018–2019 school year. Analyses included a multilevel logistic regression of school and district fidelity of implementation data, as well as school characteristics and district PBIS data. Results showed that most schools had sustained implementation after 5 years, and adequate fidelity of implementation and better team use of data for decision-making in Year 1 were the strongest predictors of sustained implementation in the 2022–2023 school year. School grade levels served and school locale were also predictive of sustained implementation. We discuss implications for how teams can improve sustained implementation of evidence-based practices through massive school disruptions.

  • writeAlizer: Generate Predicted Writing Quality Scores

    by Sterett H. Mercer

    Imports variables from 'ReaderBench' (Dascalu et al., 2018), 'Coh-Metrix' (McNamara et al., 2014), and/or 'GAMET' (Crossley et al., 2019) output files; downloads predictive scoring models described in Mercer & Cannon (2022) and Mercer et al.(2021); and generates predicted writing quality and curriculum-based measurement (McMaster & Espin, 2007) scores.


  • Factors Predicting Sustained Implementation of Tier 2 and Tier 3 Positive Behavioral Interventions and Supports

    by Angus Kittelman, Kent McIntosh, Sterett H. Mercer, Rhonda N. T. Nese, SoLing So, Heather Peshak George

    Sustained implementation of effective behavior support systems and practices is critical for improving academic and behavior outcomes for students with and without disabilities. Although implementation studies have identified variables facilitating sustained implementation of Tier 1 behavior support systems in schools, little research exists examining Tier 2 and 3 behavior support systems. The purpose of this 5-year prospective study was to identify practice, school, and district variables facilitating sustained implementation of Tiers 2 and 3 behavior support systems in schools. The sample included 646 schools implementing positive behavioral interventions and supports (PBIS) across 23 U.S. states. Using structural equation modeling, we found several key variables to be predictive of sustained implementation (sustained implementation and implementation quality) of Tier 2 (Tiers 1 and 2 fidelity in Year 1 and a general Tiers 2 and 3 sustainability factor) and Tier 3 behavior support systems (Tier 1 and Tier 3 fidelity in Year 1 and a general Tiers 2 and 3 sustainability factor). Implications for how district and school leadership teams can improve sustained implementation of Tiers 2 and 3 behavior support systems are discussed.