Antony John Kunnan

A new computer-assisted test of academic English for use at an Asian University was commissioned by administrators. The test was designed to serve both placement and diagnostic purposes. The authors and their team conceptualized, developed, and administered a scenario-based assessment with an online delivery with independent and integrated language skills tasks. The project provided many advantages: (1) the test would be locally developed by university faculty and students who would have a good understanding of the test takers and the needs of the university, (2) the test would use topics, texts, and materials and technology that are socially and culturally appropriate and sensitive to the local context, and (3) the sustainability of the test would be higher as it were cost-effective in the long run in comparison to purchasing and renewing a license for an international test. This article documents the key considerations and processes in the development of this new scenario-based test of academic English that was conceptualized and designed by faculty and students collaboratively. It also discusses the challenges involved in the implementation of such a test, including resistance from local assessment culture and high workload of language teachers.

The Duolingo English Test is a groundbreaking, digital­first, computer­adaptive measure of English language proficiency for communication and use in English­medium settings. The test measures four key English language proficiency constructs: Speaking, Writing, Reading, and Listening (SWRL), and is aligned with the Common European Framework of Reference for Languages (CEFR) proficiency levels and descriptors. As a digital­first assessment, the test uses “human­in­the­loop AI” from end to end for test security, automated item generation, and scoring of test­taker responses. This paper presents a novel theoretical assessment ecosystem for the Duolingo English Test. It is a theoretical representation of language assessment design, measurement, and test security processes, as well as the test­taker experience factors that contribute to the test validity argument and test impact. The test validity argument is constructed with a digitally­informed chain of inferences that addresses digital affordances applied to the test. The ecosystem is composed of an integrated set of complex frameworks: (1) the Language Assessment Design Framework, (2) the Expanded Evidence­Centered Design Framework, (3) the Computational Psychometrics Framework, and (4) the Test Security Framework. Test­taker experience (TTX) is a test priority throughout the test­taking pipeline, such as low cost, anytime/anywhere, and shorter testing time. The test’s expected impact is aligned with Duolingo’s social mission to lower barriers to education access and offer a secure and delightful test experience, while providing a valid, fair, and reliable test score. The ecosystem leverages principles from assessment theory, computational psychometrics, design, data science, language assessment theory, NLP/AI, and test security.

The Cambridge Handbook of Language Standardization - July 2021

An essential hypothesis of modern language assessment theory pertains to the interaction between strategy use ability (strategic competence) and second language knowledge. However, how they interact with each other is rarely explored. Drawing on relevant research in the literature, in this paper we proposed three interaction patterns (i.e., linear, quadratic, and cuboid) in which language knowledge moderates the effect of strategy use ability on L2 reading performance. A pool of 1491 nursing students were invited to respond to three instruments, each measuring language knowledge, strategy use ability, and nursing English (L2) reading ability, respectively. Student responses were first scored using multidimensional item response theory (MIRT). Next, we applied multi-layered moderation analysis (MLMA) to these MIRT-based scores to detect the hypothetical interaction patterns. The results supported the cuboid interaction pattern or, metaphorically, the pattern of an island ridge curve (IRC). Substantially, this indicated that the effect of strategy use ability on nursing English reading performance fluctuated in a down-up-down pattern with the increase of students’ language knowledge. Our study also revealed different patterns of strategy use depending on students’ language knowledge level.

This study examined the separability of domain-general and domain-specific content knowledge from Language for Specific Purposes (LSP) reading ability. A pool of 1,491 nursing students in China participated by responding to a nursing English test and a nursing knowledge test. Primary data analysis involved four steps: (a) conducting a bifactor-multidimensional item response theory model (bifactor-MIRT) analysis to establish measurement validity for the assumed domain-general factor and domain-specific factors underlying each test and to compute bifactor-MIRT direct scores; (b) transforming the bifactor-MIRT scores into composite scores; (c) conducting a confirmatory factor analysis with the composite scores to reconstruct the orginal bifactor-MIRT models, and (d) conducting a structural equation modeling analysis to explore the relationship between nursing knowledge factors (domain-general and domain-specific) and the nursing English reading factors (domain-general and domain-specific). The results showed that the domain-specific passage factors were significantly correlated with their corresponding domain-specific nursing knowledge factors and that domain-general nursing knowledge significantly predicted the variance of the domain-general reading factor. Overall, we concluded that content knowledge is inseparable from LSP reading ability. The implications for understanding LSP ability and for LSP reading test scoring are discussed.