Jiangang Hao

Assessing communication and collaboration at scale depends on a labor‐intensive task of coding communication data into categories according to different frameworks. Prior research has established that large language models (LLMs), particularly those from the GPT family, can be directly instructed with coding rubrics to code communication data and achieve accuracy comparable to human raters. However, whether the coding from LLMs perform consistently across different demographic groups, such as gender and race, remains unclear. To address this gap, we introduce three checks for evaluating subgroup consistency of LLM‐based coding, and empirically examine them based on data from three types of collaborative tasks. Our results show that LLM‐based coding perform consistently in the same way as human raters across gender or racial/ethnic groups, demonstrating the possibility of its use in large‐scale assessments of collaboration, communication, and other AI‐enabled conversation‐based assessments.

As part of this EM:IP special issue honoring Robert (Bob) J. Mislevy, this paper highlights his critical role in shaping the emerging field of computational psychometrics, which integrates computational methods from data science and artificial intelligence (AI) with core psychometric principles to address the challenges posed by complex, high‐volume multimodal data from interactive digital assessments. We discuss how Bob's vision, mentorship, and intellectual leadership helped to establish the conceptual foundations of this field, and how his foresight continues to guide the ongoing transformation of assessment in the era of AI.

Coding communication data is essential for assessing 21st-century skills such as collaboration and communication, but large-scale human coding is labor-intensive. Large language models (LLMs) such as ChatGPT offer a scalable alternative, yet their accuracy depends on both coding framework complexity and prompting strategy. Using a communication coding framework with five main categories and seventeen subcategories, we compared two prompting strategies: a hierarchical strategy that first assigns main categories and then codes subcategories, and a direct strategy that directly codes subcategories in a single step. Coding accuracy was evaluated against human coding using Cohen’s Kappa and mixed-effects logistic regression. Both strategies achieved agreement comparable to human–human reliability (overall κ ≈ 0.57–0.59). However, direct prompting consistently outperformed hierarchical prompting, yielding an approximately 18% increase in the odds of agreement. Hierarchical prompting was more susceptible to error propagation when main categories were misclassified, whereas direct prompting produced more stable subcategory coding. These results provide guidance for using LLMs to code communication data under complex coding frameworks.

Social and Emotional Learning (SEL) helps students to develop healthy identities, regulate emotions, build relationships, and make responsible decisions. Effective SEL instruction often relies on group-based activities; however, traditional classrooms without technological support make it difficult to capture the fine-grained interactions among students. As a result, teachers struggle to provide timely feedback and interventions. This paper reports on an ongoing project that leverages a computer-supported collaborative learning (CSCL) environment to implement 36 SEL lessons with fifth and sixth-grade students over the course of a semester. We describe the design and implementation of the curriculum and present preliminary findings from a pilot study. Early results indicate positive feedback from both teachers and students, along with a negative correlation between students’ off-topic chat behavior, characterized by nonsensical or “gibberish” messages, and their learning performance.

Writing is a foundational literacy skill that underpins effective communication, fosters critical thinking, facilitates learning across disciplines, and enables individuals to organize and articulate complex ideas. Consequently, writing assessment plays a vital role in evaluating language proficiency, communicative effectiveness, and analytical reasoning. The rapid advancement of large language models (LLMs) has made it increasingly easy to generate coherent, high-quality essays, raising significant concerns about the authenticity of student-submitted work. This chapter first provides an overview of the current landscape of detectors for AI-generated and AI-assisted essays, along with guidelines for their responsible use. It then presents empirical analyses to evaluate how well detectors trained on essays from one LLM generalize to identifying essays produced by other LLMs, based on essays generated in response to public GRE writing prompts. These findings provide guidance for developing and retraining detectors for practical applications.

The rapid advancement of large language models (LLMs) has enabled the generation of coherent essays, making AI‐assisted writing increasingly common in educational and professional settings. Using large‐scale empirical data, we examine and benchmark the characteristics and quality of essays generated by popular LLMs and discuss their implications for two key components of writing assessments: automated scoring and academic integrity. Our findings highlight limitations in existing automated scoring systems when applied to essays generated or heavily influenced by AI, and identify areas for improvement, including the development of new features to capture deeper thinking and recalibrating feature weights. Despite growing concerns that the increasing variety of LLMs may undermine the feasibility of detecting AI‐generated essays, our results show that detectors trained on essays generated from one model can often identify texts from others with high accuracy, suggesting that effective detection could remain manageable in practice.

The rise of large language models (LLMs) and generative AI is revolutionizing educational assessment, presenting transformative opportunities alongside critical challenges. This chapter explores these new opportunities through the lens of Evidence-Centered Design and examines the challenges along two key dimensions: adherence to the core measurement values and the growing concerns about academic integrity. The goal is to provide researchers, educators, and policymakers with a concise overview of the responsible and effective integration of AI into educational measurement.

Evaluating spoken language proficiency stands as a pivotal component in language assessment. A well-known method for cheating in spoken language tests involves reciting pre-scripted responses. This paper presents a system called the Automated Speech Similarity Detector (AutoSSD). AutoSSD leverages state-of-the-art AI transcription to convert spoken responses into texts, analyzes these transcriptions for verbatim similarities among test takers, and provides an interactive interface for a subsequent expert review. Our system has proven effective in practical operation, and our findings can help other spoken language assessments bolster their test security measures.

A frequently encountered security issue in writing tests is non-authentic text submission:test takers submit texts that are not of their own but are copies of texts prepared by some-one else. In this paper, we propose AutoESD, a human-in-the-loop and automated system todetect non-authentic texts for a large-scale writing test, and report its performance on an op-erational data set. The AutoESD system utilizes multiple automated text similarity measuresto identify suspect texts and provides an analytics-enhanced web application to help humanexpert to review the identified texts. To evaluate the performance of AutoESD, we obtained itssimilarity measures on TOEFL iBT writing responses collected from multiple remote admin-istrations and examined their distributions. The results were highly encouraging in that thedistributional characteristics of AutoESD similarity measures were effective in identifying sus-pect texts and that the measures could be computed quickly without affecting the operationalscore turnaround timeline at all.

Collaborative problem solving (CPS) is widely recognized as a critical 21st century skill. Efficiently coding communication data is an outstanding challenge in scaling up research on assessing CPS. This paper reports the findings on using ChatGPT to directly code CPS chat data by benchmarking performance across multiple datasets and coding frameworks. We found that ChatGPT-based coding outperformed human coding in tasks where the discussions were characterized by colloquial languages but fell short in tasks where the discussions dealt with specialized scientific terminology and contexts. The findings offer practical guidelines for researchers to develop strategies for efficient and scalable analysis of communication data from CPS tasks.