Thorben Jansen

Pretest–posttest–control group (PPC) effect sizes incorporate baseline information but require the pretest–posttest correlation for sampling-variance estimation, which is frequently unavailable in primary studies. This creates a practical trade-off: retaining baseline information requires an assumption about repeated-measures dependence, whereas using posttest-only effect sizes avoids this requirement but discards available baseline information. The present study used a simulation to evaluate whether missing correlations can be replaced by constant assumed values while maintaining acceptable variance calibration, and complemented the simulation with reanalyses of three published meta-analyses to illustrate the consequences for meta-analytic effects. Across 2,160 design conditions, we examined the sampling variances of two pretest-informed estimators (dppc1 and dppc2). Performance was evaluated in terms of coverage of nominal 95% confidence intervals and standard-error ratios. Across the fixed-correlation conditions, coverage declined from .97 to .83 for dppc1 and from .97 to .79 for dppc2 as the assumed correlation increased from .20 to .80, with corresponding standard-error ratios declining from 1.31 to 0.70 and from 1.32 to 0.68. Misspecification was asymmetric: overestimating the true correlation produced anti-conservative standard errors and undercoverage, whereas underestimating it generally produced conservative inference. Across the empirical examples, pooled PPC estimates were comparatively stable across correlation assumptions, whereas ignoring baseline information produced larger shifts in several pooled effects. Based on these findings, we provide practical guidelines for handling missing pretest–posttest correlations, emphasizing a hierarchical use of study-specific information, external evidence, and transparent sensitivity analyses when constant assumptions are required. An accompanying Shiny app supports these analyses: https://lliebenow.shinyapps.io/ppc-rho-sensitivity-explorer/.

Background: Argumentative writing is a key competence for academic success. Feedback can foster competence development, yet the associated outcomes show considerable heterogeneity among students. Students’ achievement goals and argumentative writing skills moderate the effectiveness of feedback in classroom settings. However, it is unclear whether these individual differences explain variations in feedback effects within AI-supported classroom contexts.Objective: We examine individual differences in the effectiveness of AI feedback on argumentative writing performance by assessing how the feedback condition (AI feedback versus generic revision prompt) interacts with students’ achievement goals (mastery, performance-approach, and performance-avoidance) and their argumentative writing skill to predict text revision quality.Method: In a classroom-based experiment, secondary school students (N = 805) wrote and revised argumentative texts. Students were randomly assigned to receive either immediate, individualized AI feedback or a generic revision prompt. The quality of the revised argumentative text was the outcome variable.Results: Mastery goal orientation moderated feedback effects. The benefit of AI feedback compared to the revision prompt was highest for students with low mastery goals. Furthermore, argumentative writing skill moderated the interaction between performance-approach goals and the feedback condition in a three-way interaction. For students with higher baseline argumentative writing skill, the benefit of AI feedback on revision quality was positively associated with a stronger performance-approach. Conversely, for students with lower baseline skills, a stronger performance-approach orientation was associated with higher revision quality in the control condition than in the AI feedback condition.Conclusion: AI feedback effects on text revision quality are conditional. Explaining differences in its effectiveness requires the simultaneous consideration of feedback conditions, students’ achievement goals, and their baseline argumentative writing skills.

Feedback is essential for enhancing student learning with complex tasks, but creating it is often time-consuming for teachers. Artificial intelligence (AI), particularly large language models, can support teachers in providing feedback. However, teachers’ use of AI depends on their perceptions of the AI-generated feedback. Drawing on the social identity theory (Tajfel & Turner, 1979), it is hypothesised that teachers prefer feedback from fellow teachers (their ingroup) to AI-generated feedback (an outgroup); this leads to lower perception ratings of AI feedback compared to teacher feedback. This study experimentally examined whether the same feedback would be perceived negatively when labelled as “AI-generated” rather than when labelled as “teacher-generated” and whether providing teachers with additional information on AI’s functionality would influence this effect. A sample of 401 English-speaking teachers read three student essays, each with two AI-generated feedback messages, randomly labelled as AI-generated or teacher-generated. They indicated their perceptions of the feedback messages regarding credibility, usefulness, and fairness, as well as their feedback preference (teacher vs. AI). The results revealed that feedback perceptions were significantly less favourable for feedback labelled as AI-generated than for feedback labelled as teacher-generated, with the majority of teachers preferring teacher-labelled feedback to AI-labelled feedback (73% vs. 27%). These findings demonstrate that teachers systematically rate identical feedback as less credible, useful, and fair when it is labelled as AI-generated rather than teacher-generated, suggesting a negative bias that may have implications for the adoption of effective AI feedback in educational practice.

By quantitatively integrating results of meta-analyses, second-order meta-analyses (SOMAs) from educational psychology address far-reaching questions of interest to adjacent scientific fields, policy, and educational practice. Conducting a SOMA, however, is time-consuming, particularly the extraction of statistical results from meta-analyses. Large Language Models (LLMs) have the potential to accelerate data extraction, as evidenced by evaluations of data extraction from primary studies. Evaluations of their accuracy in extracting results from meta-analyses to conduct SOMA are needed. We compare the accuracy of three LLMs with that of human experts extracting data from 156 educational meta-analyses investigating students’ achievement, randomly chosen from the Visible Learning database. LLMs and humans extracted effect sizes, the number of included studies, and the number of students. To assess accuracy, we established a gold standard benchmark by resolving discrepancies between LLMs and human experts through expert adjudication. Results show that the distributions of data from all coders and the standard were similar and showed no systematic bias. Accuracy reached ICCs of 0.95/0.81 for the two human experts, and 0.96/0.97/0.96 for LLMs (Gemini2.5Pro/GPT4.1/GPTo3), with percentage agreement of 86%/80% (humans) and 81%/78%/77% (LLMs). The results demonstrate that LLMs achieve data extraction accuracy comparable to that of human experts. We discuss the conditions under which data extracted by LLMs, humans, or a hybrid of both can be considered validated for use in educational SOMAs.

Feedback can play a pivotal role in learning but is only effective when students use it. However, the use of feedback varies greatly from student to student and these differences must be explained. Hence, the construct receptivity to instructional feedback (RIF) was introduced to assess students’ attitudes towards and engagement with human-provided feedback in multiple English-speaking countries. However, RIF scales have not yet been used in computer-based assessment contexts nor in German-speaking countries; their application is thus limited. This study examined an adapted and translated RIF scale using confirmatory factor analysis (CFA) and exploratory structural equation modeling (ESEM). We capitalized on data sets comprising German-speaking university ( N = 464) and secondary school ( N = 1,207) students. The bifactor-CFA model showed a good model fit and revealed a general receptivity factor alongside the four specific factors: experiential attitudes, instrumental attitudes, cognitive engagement, and behavioral engagement with feedback. Scalar measurement invariance allowed for cross-gender and educational level comparisons. The relationships with broader personality characteristics were consistent with other contexts, thus indicating discriminant validity. General receptivity, cognitive engagement, and experiential attitudes positively correlated with GPA, suggesting convergent validity. Results support the stability of the RIF scale’s adaptation and translation. Assessment implications are discussed.