CKCarmen Köhler

Carmen Köhler

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Carmen Köhler has a total of 28 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Pretest measures – To include or not to include?

    by Carmen Köhler, Johannes Hartig

  • Open Access

    KUAI-Kids: Developing a Measurement Instrument for Children’s Knowledge, Use, and Attitudes Toward Generative AI Chatbots

    by Sarah Malone, Carmen Koehler, Kristin Altmeyer, Dominik Thüs, Marina Dombrovskaia, Johannes Hartig

    The increasing presence of generative artificial intelligence in children’s lives highlights the need for age-appropriate measurement instruments to assess their knowledge, use, and atti-tudes toward these technologies. This study introduces and validates KUAI-Kids, a question-naire on ChatGPT knowledge, use, and attitudes for children in middle childhood and early adolescence (ages 7 to 14; N = 310), based on a previously validated adult scale but with sub-stantially revised and extended items for children. Confirmatory factor analyses supported uni-dimensional structures for the Knowledge and Use scales, both showing good model fit and internal consistency. For the Attitude scale, the originally assumed unidimensional structure was not supported. Instead, factor analyses identified two distinct dimensions, Enthusiastic and Cautious Attitudes toward ChatGPT. Both Attitude subscales demonstrated good model fit and reliability. Latent correlations revealed expected associations: Knowledge and Use were mod-erately positively correlated; Enthusiastic Attitudes were strongly associated with Use, while a moderately strong correlation was observed between Cautious Attitudes and Knowledge. Ex-ploratory analyses indicated higher Enthusiastic Attitudes in boys and higher Knowledge, Use, and Cautious Attitudes scores in older students. The KUAI-Kids offers a psychometrically sound tool to study children’s engagement with AI chatbots and to support AI literacy assess-ment, the evaluation of educational interventions, and evidence-based development of AI-related educational policies. Limitations include the rapidly evolving nature of generative AI technologies and the fact that many children in the sample had little or no direct experience with ChatGPT.

  • Open Access

    Rating procedures to evaluate generic critical online reasoning in an open Internet environment

    by Johannes Hartig, Philine Drake, Jannick Illmann, Carmen Köhler, Anika Kohmer, Alisa Stroganova

    University students regularly rely on Internet resources for academic work, requiring the ability to critically evaluate search results, identify relevant information, and draw well-founded conclusions. We refer to these skills as critical online reasoning (COR). This paper presents methods to assess Generic Critical Online Reasoning (GEN-COR) in an open Internet environment, focusing on online information acquisition (OIA) and reasoning based on evidence and synthesis (REAS). Based on data from two tasks, we illustrate the principles of task construction and the development of criteria to evaluate task performance. Using data from 376 first-year students from different fields, we examine inter-rater reliability, indicator difficulty, and the factorial structure of OIA and REAS. Results show that human ratings reliably assess GEN-COR performance and that OIA and REAS are distinct, although surprisingly uncorrelated dimensions. Further research is needed to validate these findings across tasks and to integrate critical information evaluation (CIE) skills.


  • Open Access

    Investigating Data Sharing Practices in Teaching Effectiveness Research

    by Talha Sajjad, Johannes Hartig, Thomas Lösch, Carmen Köhler

    Sharing research data promotes transparency and cumulative knowledge building. This is especially relevant in teaching effectiveness research, where acquiring classroom data is challenging. Despite the benefits, previous studies indicate low data availability in educational research. This study examines data sharing practices in 167 studies, acquired via a two-step process: first, identifying meta-analyses in teaching effectiveness research, and second, scanning them for articles meeting specific criteria (longitudinal, peer-reviewed, primary data, publication year). For each article, we checked for a repository link and contacted corresponding authors if none was provided. Only 13 publications included a repository link. Among the contacted researchers, one-third had outdated contact details. Of the remaining researchers, 36 responded, with 9 agreeing to share their data. The overall availability rate was 13.2%, with the odds of data-sharing decreasing by 19% annually post-publication. Our results show the need to archive data in repositories, as its availability upon request remains limited.

  • Open Access

    Statistical decisions when modelling effects of teaching quality

    by Carmen Köhler, Benjamin Herbert, Anna-Katharina Praetorius


  • Open Access

    ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude

    by Carmen Köhler, Johannes Hartig

    Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). N = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.


  • Open Access

    DIF Detection for Multiple Groups: Comparing Three‐Level GLMMs and Multiple‐Group IRT Models

    by Carmen Köhler, Lale Khorramdel, Artur Pokropek, Johannes Hartig

    For assessment scales applied to different groups (e.g., students from different states; patients in different countries), multigroup differential item functioning (MG‐DIF) needs to be evaluated in order to ensure that respondents with the same trait level but from different groups have equal response probabilities on a particular item. The current study compares two approaches for DIF detection: a multiple‐group item response theory (MG‐IRT) model and a generalized linear mixed model (GLMM). In the MG‐IRT model approach, item parameters are constrained to be equal across groups and DIF is evaluated for each item in each group. In the GLMM, groups are treated as random, and item difficulties are modeled as correlated random effects with a joint multivariate normal distribution. Its nested structure allows the estimation of item difficulty variances and covariances at the group level. We use an excerpt from the PISA 2015 reading domain as an exemplary empirical investigation, and conduct a simulation study to compare the performance of the two approaches. Results from the empirical investigation show that the detection of countries with DIF is similar in both approaches. Results from the simulation study confirm this finding and indicate slight advantages of the MG‐IRT model approach.


  • Open Access

    Practical significance of item misfit and its manifestations in constructs assessed in large-scale studies

    by Katharina Fährmann, Carmen Köhler, Johannes Hartig, Jörg-Henrik Heine

    When scaling psychological tests with methods of item response theory it is necessary to investigate to what extent the responses correspond to the model predictions. In addition to the statistical evaluation of item misfit, the question arises as to its practical significance. Although item removal is undesirable for several reasons, its practical consequences are rarely investigated and focus mostly on main survey data with pre-selected items. In this paper, we identify criteria to evaluate practical significance and discuss them with respect to various types of assessments and their particular purposes. We then demonstrate the practical consequences of item misfit using two data examples from the German PISA 2018 field trial study: one with cognitive data and one with non-cognitive/metacognitive data. For the former, we scale the data under the GPCM with and without the inclusion of misfitting items, and investigate how this influences the trait distribution and the allocation to reading competency levels. For non-cognitive/metacognitive data, we explore the effect of excluding misfitting items on estimated gender differences. Our results indicate minor practical consequences for person allocation and no changes in the estimated gender-difference effects.


  • Der Beitrag der Kompetenztheorie für die Fachdidaktiken

    by Alexander Naumann, Carmen Köhler, Johannes Hartig