Ben Naismith
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Research Interests
Ben Naismith has a total of 23 co-authors. Here are the top co-authors by number of shared publications:
- Alan Juffs
University of Pittsburgh
5 shared publications - Geoffrey T. LaFlair
Duolingo, USA
3 shared publications - Ramsey Cardwell
Duolingo, USA
3 shared publications - Jill Burstein
Duolingo, Inc., USA
2 shared publications - Kevin P. Yancey
Duolingo English Test Duolingo, Inc
2 shared publications - Maria Kostromitina
Duolingo, Inc., Pittsburgh, PA 15206, USA
2 shared publications
Latest Publications
- Open Access
- 20 April 2026
- Published by SAGE Publications in Language Testing
A Systematic Review of Test Component Ordering with Implications for Language Assessment
How to optimize ordering the parts of a language test (e.g., items, tasks, and sections) is often overlooked or assumed to be self-evident. However, ordering choices are an essential component of test design as they may impact test takers’ objective performance, perceived performance, or affective states. In this paper, we report on a systematic review of test component ordering research, focusing on 88 studies from 1933 to 2023. We provide a narrative synthesis, describing typical outcome variables (e.g., test-taker performance), independent variables (e.g., order of difficulty), and mediating variables (e.g., language proficiency). Key findings, mostly from higher educational contexts, indicate that easy-to-hard ordering may lead to better performance, though these effects are mediated by various test-taker characteristics, especially anxiety and proficiency. While section ordering by language skills is common practice in language testing, there is scant empirical support for this approach, or for organizing tests by content or format. We discuss the implications of these findings for language test developers and suggest avenues for future research, particularly the need for more studies on ordering effects in language assessment contexts.
- 20 April 2026
- Published by SAGE Publications in Language Testing
Open AccessA Systematic Review of Test Component Ordering with Implications for Language Assessment
How to optimize ordering the parts of a language test (e.g., items, tasks, and sections) is often overlooked or assumed to be self-evident. However, ordering choices are an essential component of test design as they may impact test takers’ objective performance, perceived performance, or affective states. In this paper, we report on a systematic review of test component ordering research, focusing on 88 studies from 1933 to 2023. We provide a narrative synthesis, describing typical outcome variables (e.g., test-taker performance), independent variables (e.g., order of difficulty), and mediating variables (e.g., language proficiency). Key findings, mostly from higher educational contexts, indicate that easy-to-hard ordering may lead to better performance, though these effects are mediated by various test-taker characteristics, especially anxiety and proficiency. While section ordering by language skills is common practice in language testing, there is scant empirical support for this approach, or for organizing tests by content or format. We discuss the implications of these findings for language test developers and suggest avenues for future research, particularly the need for more studies on ordering effects in language assessment contexts.
- Open Access
- 12 August 2025
- Published by Wiley in Review of Education
- Vol. 13 (2)
Beyond GPA and language proficiency: A systematic literature review of international students' academic success factors
The rapid increase in international students, many of whom study in a second language (L2), has driven extensive research that examines the contribution of L2 proficiency to students' academic success. However, focusing solely on L2 proficiency as captured by language test scores provides a limited perspective, ignoring the many factors that contribute to academic achievement. This systematic literature review aims to broaden our understanding of this topic by exploring additional factors that contribute to the academic success of international students. Analysing 54 peer‐reviewed studies from 1990 to 2023, the review highlights that while outcome measures of success remain largely focused on grade point average (GPA), researchers are increasingly investigating predictors other than language proficiency, such as (1) background characteristics (e.g. gender), (2) individual differences (e.g. personality), (3) previous educational experiences (e.g. high school grades), (4) current educational experiences (e.g. familiarity with the learning environment), and (5) current social experiences (e.g. sense of belonging), which display varying effect sizes on international student success. Findings from this study have implications for academic stakeholders, suggesting a more holistic approach to understanding and supporting the academic success of international students. Context and implications: Rationale for this study: This systematic review addresses a critical gap in understanding international student success. While existing research has focused heavily on language proficiency and GPA, this study offers a more holistic view of success factors, recognising that the international student population faces complex challenges beyond language barriers, and that student success is measured not only via GPA. Why the new findings matter: The findings challenge the narrow focus on GPA and language proficiency as prevalent metrics of international student success by revealing a broader range of factors that shape international student success. In addition, the study explores the quality of scholarship in the domain. Implications for practitioners, policy makers, and researchers: For researchers, this study highlights the need for more transparent research designs and reporting practices when investigating predictors of international student success. It is also recommended that research should expand into understudied areas in this domain, particularly looking at international education in non‐English speaking destinations and at the graduate level. For faculty, findings emphasise the importance of implementing culturally responsive teaching practices and creating inclusive learning environments. Finally, for university admissions, the study indicates that relying solely on language proficiency scores provides an incomplete picture of potential success, suggesting a more holistic admissions approach.
- 12 August 2025
- Published by Wiley in Review of Education
- Vol. 13 (2)
Open AccessBeyond GPA and language proficiency: A systematic literature review of international students' academic success factors
The rapid increase in international students, many of whom study in a second language (L2), has driven extensive research that examines the contribution of L2 proficiency to students' academic success. However, focusing solely on L2 proficiency as captured by language test scores provides a limited perspective, ignoring the many factors that contribute to academic achievement. This systematic literature review aims to broaden our understanding of this topic by exploring additional factors that contribute to the academic success of international students. Analysing 54 peer‐reviewed studies from 1990 to 2023, the review highlights that while outcome measures of success remain largely focused on grade point average (GPA), researchers are increasingly investigating predictors other than language proficiency, such as (1) background characteristics (e.g. gender), (2) individual differences (e.g. personality), (3) previous educational experiences (e.g. high school grades), (4) current educational experiences (e.g. familiarity with the learning environment), and (5) current social experiences (e.g. sense of belonging), which display varying effect sizes on international student success. Findings from this study have implications for academic stakeholders, suggesting a more holistic approach to understanding and supporting the academic success of international students. Context and implications: Rationale for this study: This systematic review addresses a critical gap in understanding international student success. While existing research has focused heavily on language proficiency and GPA, this study offers a more holistic view of success factors, recognising that the international student population faces complex challenges beyond language barriers, and that student success is measured not only via GPA. Why the new findings matter: The findings challenge the narrow focus on GPA and language proficiency as prevalent metrics of international student success by revealing a broader range of factors that shape international student success. In addition, the study explores the quality of scholarship in the domain. Implications for practitioners, policy makers, and researchers: For researchers, this study highlights the need for more transparent research designs and reporting practices when investigating predictors of international student success. It is also recommended that research should expand into understudied areas in this domain, particularly looking at international education in non‐English speaking destinations and at the graduate level. For faculty, findings emphasise the importance of implementing culturally responsive teaching practices and creating inclusive learning environments. Finally, for university admissions, the study indicates that relying solely on language proficiency scores provides an incomplete picture of potential success, suggesting a more holistic admissions approach.
- Open Access
- 14 July 2025
- Published by Wiley in Language Learning
- Vol. 75 (S1) , 170-203
Developing an Automatic Pronunciation Scorer: Aligning Speech Evaluation Models and Applied Linguistics Constructs
Globalization and increases in the numbers of English language learners have led to a growing demand for English proficiency assessments of spoken language. In this paper, we describe the development of an automatic pronunciation scorer built on state‐of‐the‐art deep neural network models. The model is trained on a bespoke human‐rated dataset that reflects current perspectives on pronunciation and intelligibility. The new scorer is evaluated along three criteria: How well it explains expert human ratings, how it compares to other state‐of‐the‐art automatic pronunciation scorers in explaining expert human ratings, and the extent to which it exhibits bias toward different groups of test takers. Results indicate that the proposed scorer shows strong positive correlations with expert human ratings and outperforms other scorers. However, the scorer shows some bias related to audio quality and language family groups. We conclude with future directions for mitigating bias and argue that this scorer holds potential for use in operational settings.
- 14 July 2025
- Published by Wiley in Language Learning
- Vol. 75 (S1) , 170-203
Open AccessDeveloping an Automatic Pronunciation Scorer: Aligning Speech Evaluation Models and Applied Linguistics Constructs
Globalization and increases in the numbers of English language learners have led to a growing demand for English proficiency assessments of spoken language. In this paper, we describe the development of an automatic pronunciation scorer built on state‐of‐the‐art deep neural network models. The model is trained on a bespoke human‐rated dataset that reflects current perspectives on pronunciation and intelligibility. The new scorer is evaluated along three criteria: How well it explains expert human ratings, how it compares to other state‐of‐the‐art automatic pronunciation scorers in explaining expert human ratings, and the extent to which it exhibits bias toward different groups of test takers. Results indicate that the proposed scorer shows strong positive correlations with expert human ratings and outperforms other scorers. However, the scorer shows some bias related to audio quality and language family groups. We conclude with future directions for mitigating bias and argue that this scorer holds potential for use in operational settings.
- Open Access
- 5 February 2025
- Published by Cambridge University Press (CUP) in Studies in Second Language Acquisition
- Vol. 47 (1) , 336-360
The impact of collocational proficiency features on expert ratings of L2 English learners’ writing
Lexical proficiency is a multifaceted phenomenon that greatly impacts human judgments of writing quality. However, the importance of collocations’ contribution to proficiency assessment has received less attention than that of single words, despite collocations’ essential role in language production. This study, therefore, investigated how aspects of collocational proficiency affect the ratings that examiners give to English learner essays. To do so, collocational features related to sophistication and accuracy were manipulated in a set of argumentative essays. Examiners then rated the texts and provided rationales for their choices. The findings revealed that the use of lower-frequency words significantly and positively impacted the experts’ ratings. When used as part of collocations, such words then provided a small yet significant additional boost to ratings. Notably, there was no significant effect for increased collocational accuracy. These findings suggest that low-frequency words within collocations are particularly salient to examiners and deserving of pedagogic focus.
- 5 February 2025
- Published by Cambridge University Press (CUP) in Studies in Second Language Acquisition
- Vol. 47 (1) , 336-360
Open AccessThe impact of collocational proficiency features on expert ratings of L2 English learners’ writing
Lexical proficiency is a multifaceted phenomenon that greatly impacts human judgments of writing quality. However, the importance of collocations’ contribution to proficiency assessment has received less attention than that of single words, despite collocations’ essential role in language production. This study, therefore, investigated how aspects of collocational proficiency affect the ratings that examiners give to English learner essays. To do so, collocational features related to sophistication and accuracy were manipulated in a set of argumentative essays. Examiners then rated the texts and provided rationales for their choices. The findings revealed that the use of lower-frequency words significantly and positively impacted the experts’ ratings. When used as part of collocations, such words then provided a small yet significant additional boost to ratings. Notably, there was no significant effect for increased collocational accuracy. These findings suggest that low-frequency words within collocations are particularly salient to examiners and deserving of pedagogic focus.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 26-37
Identifying and analyzing ‘noisy’ spelling errors in a second language corpus
Alan Juffs, Ben Naismith. Proceedings of the Tenth Workshop on Noisy and User-generated Text. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 26-37
Open AccessIdentifying and analyzing ‘noisy’ spelling errors in a second language corpus
Alan Juffs, Ben Naismith. Proceedings of the Tenth Workshop on Noisy and User-generated Text. 2025.
- Open Access
- 30 October 2024
- Published by Elsevier in Assessing Writing
- Vol. 62
The impact of task duration on the scoring of independent writing responses of adult L2-English writers
- 30 October 2024
- Published by Elsevier in Assessing Writing
- Vol. 62
Open AccessThe impact of task duration on the scoring of independent writing responses of adult L2-English writers
- 3 January 2024
- Published by Taylor & Francis
- p. 173-196
“I'm not a walking dictionary”
Language teachers’ self-efficacy (LTSE) beliefs encompass the many ways in which language teachers view their own capabilities to support language learning. Importantly, these means are task-, domain-, and context-specific, interacting with language teachers’ beliefs about their own language proficiency and their own language awareness. To date, there has been extensive research into LTSE beliefs and teacher cognition more generally, but specific language domains such as vocabulary have not received due attention. To help address this gap, this chapter reports on a mixed-methods investigation of LTSE beliefs about teaching vocabulary and the relationship of these beliefs to lexical awareness. Quantitative data were collected through an online survey and were supplemented with qualitative data from semi-structured interviews. The findings reveal that confidence in presenting/practising vocabulary and what we termed “responsively using teacher lexical knowledge” were the greatest predictors of overall vocabulary teaching confidence. Other trends that emerged related to beliefs about pre-teaching vocabulary, beliefs about using learners’ first language, and teachers’ awareness of multi-word lexical items. These findings have implications for teacher preparation, suggesting aspects of vocabulary instruction which may be currently underserved, uncovering characteristics of confident vocabulary teachers that can be emulated, and highlighting issues in vocabulary instruction which remain controversial.
- 3 January 2024
- Published by Taylor & Francis
- p. 173-196
“I'm not a walking dictionary”
Language teachers’ self-efficacy (LTSE) beliefs encompass the many ways in which language teachers view their own capabilities to support language learning. Importantly, these means are task-, domain-, and context-specific, interacting with language teachers’ beliefs about their own language proficiency and their own language awareness. To date, there has been extensive research into LTSE beliefs and teacher cognition more generally, but specific language domains such as vocabulary have not received due attention. To help address this gap, this chapter reports on a mixed-methods investigation of LTSE beliefs about teaching vocabulary and the relationship of these beliefs to lexical awareness. Quantitative data were collected through an online survey and were supplemented with qualitative data from semi-structured interviews. The findings reveal that confidence in presenting/practising vocabulary and what we termed “responsively using teacher lexical knowledge” were the greatest predictors of overall vocabulary teaching confidence. Other trends that emerged related to beliefs about pre-teaching vocabulary, beliefs about using learners’ first language, and teachers’ awareness of multi-word lexical items. These findings have implications for teacher preparation, suggesting aspects of vocabulary instruction which may be currently underserved, uncovering characteristics of confident vocabulary teachers that can be emulated, and highlighting issues in vocabulary instruction which remain controversial.
- 30 June 2023
- Published by Springer Nature
- p. 701-707
Ensuring Fairness of Human- and AI-Generated Test Items
- 30 June 2023
- Published by Springer Nature
- p. 701-707
Ensuring Fairness of Human- and AI-Generated Test Items