Carmen Köhler
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Carmen Köhler has a total of 28 co-authors. Here are the top co-authors by number of shared publications:
- Johannes Hartig
DIPF | Leibniz Institute for Research and Information in Education
10 shared publications Johannes Hartig
Leibniz Institute for Research and Information in Education
4 shared publications- Alexander Naumann
DIPF, Leibniz-Institut für Bildungsforschung und Bildungsinformation
2 shared publications - Alexander Robitzsch
IPN—Leibniz Institute for Science and Mathematics Education, Olshausenstraße 62, 24118 Kiel, Germany
2 shared publications Katharina Fährmann
DIPF Leibniz Institute for Research and Information in Education, Frankfurt, Germany
2 shared publicationsAlisa Stroganova
Institute of Medical Education, Ludwig Maximilian University of Munich, Munich, Germany
1 shared publication
Latest Publications
- Open Access
- 21 October 2026
- Published by Elsevier in Learning and Instruction
- Vol. 107
Pretest measures – To include or not to include?
- 21 October 2026
- Published by Elsevier in Learning and Instruction
- Vol. 107
Open AccessPretest measures – To include or not to include?
- Open Access
- 30 August 2026
- Published by Center for Open Science in PsyArXiv
KUAI-Kids: Developing a Measurement Instrument for Children’s Knowledge, Use, and Attitudes Toward Generative AI Chatbots
The increasing presence of generative artificial intelligence in children’s lives highlights the need for age-appropriate measurement instruments to assess their knowledge, use, and atti-tudes toward these technologies. This study introduces and validates KUAI-Kids, a question-naire on ChatGPT knowledge, use, and attitudes for children in middle childhood and early adolescence (ages 7 to 14; N = 310), based on a previously validated adult scale but with sub-stantially revised and extended items for children. Confirmatory factor analyses supported uni-dimensional structures for the Knowledge and Use scales, both showing good model fit and internal consistency. For the Attitude scale, the originally assumed unidimensional structure was not supported. Instead, factor analyses identified two distinct dimensions, Enthusiastic and Cautious Attitudes toward ChatGPT. Both Attitude subscales demonstrated good model fit and reliability. Latent correlations revealed expected associations: Knowledge and Use were mod-erately positively correlated; Enthusiastic Attitudes were strongly associated with Use, while a moderately strong correlation was observed between Cautious Attitudes and Knowledge. Ex-ploratory analyses indicated higher Enthusiastic Attitudes in boys and higher Knowledge, Use, and Cautious Attitudes scores in older students. The KUAI-Kids offers a psychometrically sound tool to study children’s engagement with AI chatbots and to support AI literacy assess-ment, the evaluation of educational interventions, and evidence-based development of AI-related educational policies. Limitations include the rapidly evolving nature of generative AI technologies and the fact that many children in the sample had little or no direct experience with ChatGPT.
- 30 August 2026
- Published by Center for Open Science in PsyArXiv
Open AccessKUAI-Kids: Developing a Measurement Instrument for Children’s Knowledge, Use, and Attitudes Toward Generative AI Chatbots
The increasing presence of generative artificial intelligence in children’s lives highlights the need for age-appropriate measurement instruments to assess their knowledge, use, and atti-tudes toward these technologies. This study introduces and validates KUAI-Kids, a question-naire on ChatGPT knowledge, use, and attitudes for children in middle childhood and early adolescence (ages 7 to 14; N = 310), based on a previously validated adult scale but with sub-stantially revised and extended items for children. Confirmatory factor analyses supported uni-dimensional structures for the Knowledge and Use scales, both showing good model fit and internal consistency. For the Attitude scale, the originally assumed unidimensional structure was not supported. Instead, factor analyses identified two distinct dimensions, Enthusiastic and Cautious Attitudes toward ChatGPT. Both Attitude subscales demonstrated good model fit and reliability. Latent correlations revealed expected associations: Knowledge and Use were mod-erately positively correlated; Enthusiastic Attitudes were strongly associated with Use, while a moderately strong correlation was observed between Cautious Attitudes and Knowledge. Ex-ploratory analyses indicated higher Enthusiastic Attitudes in boys and higher Knowledge, Use, and Cautious Attitudes scores in older students. The KUAI-Kids offers a psychometrically sound tool to study children’s engagement with AI chatbots and to support AI literacy assess-ment, the evaluation of educational interventions, and evidence-based development of AI-related educational policies. Limitations include the rapidly evolving nature of generative AI technologies and the fact that many children in the sample had little or no direct experience with ChatGPT.
- Open Access
- 6 May 2026
- Published by Elsevier in Studies in Educational Evaluation
- Vol. 89
How should pretest measures be included in multilevel models when examining the effects of teacher or school variables on learning?
- 6 May 2026
- Published by Elsevier in Studies in Educational Evaluation
- Vol. 89
Open AccessHow should pretest measures be included in multilevel models when examining the effects of teacher or school variables on learning?
- Open Access
- 10 February 2026
- Published by Springer Nature in Zeitschrift für Erziehungswissenschaft
- Vol. 29 (1) , 19-44
Rating procedures to evaluate generic critical online reasoning in an open Internet environment
University students regularly rely on Internet resources for academic work, requiring the ability to critically evaluate search results, identify relevant information, and draw well-founded conclusions. We refer to these skills as critical online reasoning (COR). This paper presents methods to assess Generic Critical Online Reasoning (GEN-COR) in an open Internet environment, focusing on online information acquisition (OIA) and reasoning based on evidence and synthesis (REAS). Based on data from two tasks, we illustrate the principles of task construction and the development of criteria to evaluate task performance. Using data from 376 first-year students from different fields, we examine inter-rater reliability, indicator difficulty, and the factorial structure of OIA and REAS. Results show that human ratings reliably assess GEN-COR performance and that OIA and REAS are distinct, although surprisingly uncorrelated dimensions. Further research is needed to validate these findings across tasks and to integrate critical information evaluation (CIE) skills.
- 10 February 2026
- Published by Springer Nature in Zeitschrift für Erziehungswissenschaft
- Vol. 29 (1) , 19-44
Open AccessRating procedures to evaluate generic critical online reasoning in an open Internet environment
University students regularly rely on Internet resources for academic work, requiring the ability to critically evaluate search results, identify relevant information, and draw well-founded conclusions. We refer to these skills as critical online reasoning (COR). This paper presents methods to assess Generic Critical Online Reasoning (GEN-COR) in an open Internet environment, focusing on online information acquisition (OIA) and reasoning based on evidence and synthesis (REAS). Based on data from two tasks, we illustrate the principles of task construction and the development of criteria to evaluate task performance. Using data from 376 first-year students from different fields, we examine inter-rater reliability, indicator difficulty, and the factorial structure of OIA and REAS. Results show that human ratings reliably assess GEN-COR performance and that OIA and REAS are distinct, although surprisingly uncorrelated dimensions. Further research is needed to validate these findings across tasks and to integrate critical information evaluation (CIE) skills.
- Open Access
- 31 January 2026
- Published by SAGE Publications in AERA Open
- Vol. 12
Investigating Data Sharing Practices in Teaching Effectiveness Research
Sharing research data promotes transparency and cumulative knowledge building. This is especially relevant in teaching effectiveness research, where acquiring classroom data is challenging. Despite the benefits, previous studies indicate low data availability in educational research. This study examines data sharing practices in 167 studies, acquired via a two-step process: first, identifying meta-analyses in teaching effectiveness research, and second, scanning them for articles meeting specific criteria (longitudinal, peer-reviewed, primary data, publication year). For each article, we checked for a repository link and contacted corresponding authors if none was provided. Only 13 publications included a repository link. Among the contacted researchers, one-third had outdated contact details. Of the remaining researchers, 36 responded, with 9 agreeing to share their data. The overall availability rate was 13.2%, with the odds of data-sharing decreasing by 19% annually post-publication. Our results show the need to archive data in repositories, as its availability upon request remains limited.
- 31 January 2026
- Published by SAGE Publications in AERA Open
- Vol. 12
Open AccessInvestigating Data Sharing Practices in Teaching Effectiveness Research
Sharing research data promotes transparency and cumulative knowledge building. This is especially relevant in teaching effectiveness research, where acquiring classroom data is challenging. Despite the benefits, previous studies indicate low data availability in educational research. This study examines data sharing practices in 167 studies, acquired via a two-step process: first, identifying meta-analyses in teaching effectiveness research, and second, scanning them for articles meeting specific criteria (longitudinal, peer-reviewed, primary data, publication year). For each article, we checked for a repository link and contacted corresponding authors if none was provided. Only 13 publications included a repository link. Among the contacted researchers, one-third had outdated contact details. Of the remaining researchers, 36 responded, with 9 agreeing to share their data. The overall availability rate was 13.2%, with the odds of data-sharing decreasing by 19% annually post-publication. Our results show the need to archive data in repositories, as its availability upon request remains limited.
- Open Access
- 3 May 2025
- Published by Taylor & Francis in Educational Studies
- p. 1-28
Statistical decisions when modelling effects of teaching quality
- 3 May 2025
- Published by Taylor & Francis in Educational Studies
- p. 1-28
Open AccessStatistical decisions when modelling effects of teaching quality
- Open Access
- 1 October 2024
- Published by Bastas Publications in Contemporary Educational Technology
- Vol. 16 (4) , ep528
ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude
Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). N = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.
- 1 October 2024
- Published by Bastas Publications in Contemporary Educational Technology
- Vol. 16 (4) , ep528
Open AccessChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude
Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). N = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.
- Open Access
- 14 February 2024
- Published by Wiley in Journal of Educational Measurement
- Vol. 61 (2) , 325-344
DIF Detection for Multiple Groups: Comparing Three‐Level GLMMs and Multiple‐Group IRT Models
For assessment scales applied to different groups (e.g., students from different states; patients in different countries), multigroup differential item functioning (MG‐DIF) needs to be evaluated in order to ensure that respondents with the same trait level but from different groups have equal response probabilities on a particular item. The current study compares two approaches for DIF detection: a multiple‐group item response theory (MG‐IRT) model and a generalized linear mixed model (GLMM). In the MG‐IRT model approach, item parameters are constrained to be equal across groups and DIF is evaluated for each item in each group. In the GLMM, groups are treated as random, and item difficulties are modeled as correlated random effects with a joint multivariate normal distribution. Its nested structure allows the estimation of item difficulty variances and covariances at the group level. We use an excerpt from the PISA 2015 reading domain as an exemplary empirical investigation, and conduct a simulation study to compare the performance of the two approaches. Results from the empirical investigation show that the detection of countries with DIF is similar in both approaches. Results from the simulation study confirm this finding and indicate slight advantages of the MG‐IRT model approach.
- 14 February 2024
- Published by Wiley in Journal of Educational Measurement
- Vol. 61 (2) , 325-344
Open AccessDIF Detection for Multiple Groups: Comparing Three‐Level GLMMs and Multiple‐Group IRT Models
For assessment scales applied to different groups (e.g., students from different states; patients in different countries), multigroup differential item functioning (MG‐DIF) needs to be evaluated in order to ensure that respondents with the same trait level but from different groups have equal response probabilities on a particular item. The current study compares two approaches for DIF detection: a multiple‐group item response theory (MG‐IRT) model and a generalized linear mixed model (GLMM). In the MG‐IRT model approach, item parameters are constrained to be equal across groups and DIF is evaluated for each item in each group. In the GLMM, groups are treated as random, and item difficulties are modeled as correlated random effects with a joint multivariate normal distribution. Its nested structure allows the estimation of item difficulty variances and covariances at the group level. We use an excerpt from the PISA 2015 reading domain as an exemplary empirical investigation, and conduct a simulation study to compare the performance of the two approaches. Results from the empirical investigation show that the detection of countries with DIF is similar in both approaches. Results from the simulation study confirm this finding and indicate slight advantages of the MG‐IRT model approach.
- Open Access
- 12 July 2022
- Published by Springer Nature in Large-scale Assessments in Education
- Vol. 10 (1) , 1-21
Practical significance of item misfit and its manifestations in constructs assessed in large-scale studies
When scaling psychological tests with methods of item response theory it is necessary to investigate to what extent the responses correspond to the model predictions. In addition to the statistical evaluation of item misfit, the question arises as to its practical significance. Although item removal is undesirable for several reasons, its practical consequences are rarely investigated and focus mostly on main survey data with pre-selected items. In this paper, we identify criteria to evaluate practical significance and discuss them with respect to various types of assessments and their particular purposes. We then demonstrate the practical consequences of item misfit using two data examples from the German PISA 2018 field trial study: one with cognitive data and one with non-cognitive/metacognitive data. For the former, we scale the data under the GPCM with and without the inclusion of misfitting items, and investigate how this influences the trait distribution and the allocation to reading competency levels. For non-cognitive/metacognitive data, we explore the effect of excluding misfitting items on estimated gender differences. Our results indicate minor practical consequences for person allocation and no changes in the estimated gender-difference effects.
- 12 July 2022
- Published by Springer Nature in Large-scale Assessments in Education
- Vol. 10 (1) , 1-21
Open AccessPractical significance of item misfit and its manifestations in constructs assessed in large-scale studies
When scaling psychological tests with methods of item response theory it is necessary to investigate to what extent the responses correspond to the model predictions. In addition to the statistical evaluation of item misfit, the question arises as to its practical significance. Although item removal is undesirable for several reasons, its practical consequences are rarely investigated and focus mostly on main survey data with pre-selected items. In this paper, we identify criteria to evaluate practical significance and discuss them with respect to various types of assessments and their particular purposes. We then demonstrate the practical consequences of item misfit using two data examples from the German PISA 2018 field trial study: one with cognitive data and one with non-cognitive/metacognitive data. For the former, we scale the data under the GPCM with and without the inclusion of misfitting items, and investigate how this influences the trait distribution and the allocation to reading competency levels. For non-cognitive/metacognitive data, we explore the effect of excluding misfitting items on estimated gender differences. Our results indicate minor practical consequences for person allocation and no changes in the estimated gender-difference effects.
- 6 July 2022
- Published by Springer Nature
- p. 47-60
Der Beitrag der Kompetenztheorie für die Fachdidaktiken
- 6 July 2022
- Published by Springer Nature
- p. 47-60
Der Beitrag der Kompetenztheorie für die Fachdidaktiken