Tineke Brunfaut

This study investigates how second language (L2) writers’ textual features vary across multimodal integrated tasks and how these features relate to raters’ evaluations of performance. Reflecting the growing emphasis on multimodality in language use, our research focused on viewing-to-write tasks that combine auditory, visual, and written input. A total of 133 intermediate to advanced L2 English learners completed two task types: (1) viewing-to-describe, where participants watched a recording explaining and visually depicting a production process before writing a description; and (2) viewing-to-compare-and-contrast, where learners viewed a videocast of two experts discussing a topic, supported by visuals, and then produced a comparative report. Performances were evaluated using a purpose-developed rating scale, and the written texts were analyzed for linguistic features, including accuracy, cohesion, lexical diversity and sophistication, and syntactic and lexical complexity. Results showed that these linguistic indices explained a substantial proportion of variance in performance scores. Although the overall use of linguistic features did not differ significantly between task types, their relative contribution to scores varied. The findings have important implications for using multimodal integrated tasks in task-based language teaching and assessment.

Despite general recognition that, in human-led item writing, training is key to producing good-quality tests, there is little empirical research on what constitutes good practice in item-writing training for language test development. This has led to a lack of evidence-based guidance for those who (plan to) organise item-writing training. To help address this research gap, this study explored the perceptions of 25 novice item writers on the usefulness of a three-month, online induction item-writing training course. Views were collected via four feedback questionnaires administered at fixed points throughout the course and in semi-structured interviews conducted on course completion. Findings showed that participants particularly valued a clear bite-size course structure, extensive item-writing practice, timely and detailed tutor feedback, and regular opportunities for peer collaboration. Combining language testing theory with item-writing practice was also viewed as beneficial for learning. Participants held mixed views, however, on the platforms used for course delivery. Based on the findings, practical recommendations are proposed for how training for human-led item-writing can be usefully structured and delivered.

Comparative Judgement (CJ) is an evaluation method, typically conducted online, whereby a rank order is constructed, and scores calculated, from judges’ pairwise comparisons of performances. CJ has been researched in various educational contexts, though only rarely in English as a Foreign Language (EFL) writing settings, and is generally agreed to be a reliable method of evaluating performances. This study extends the CJ research base to young learner EFL writing contexts and innovates CJ procedures with a novel dimension-based approach. Twenty-seven Austrian EFL educators evaluated 300 young learners’ EFL scripts (addressing two task types) from a national examination, using three scoring methods: standard CJ (holistic), CJ by dimensions (our new criteria-based method), and the exam’s conventional analytic rating. It was found that both holistic CJ and our dimension-based CJ were reliable methods of evaluating young learners’ EFL scripts. Experienced EFL teachers who also have experience with using marking schemes proved to be reliable CJ judges. Moreover, despite the preference of some for the more familiar analytic rating method, teachers displayed higher reliability and shorter decision-making times when using CJ. Benefits of dimension-based CJ for reliable and economical scoring of large-scale young learner EFL writing scripts, and the potential for positive washback, are discussed.

Assessing integrated reading-into-writing task performances is known to be challenging, and analytic rating scales have been found to better facilitate the scoring of these performances than other common types of rating scales. However, little is known about how specific operationalizations of the reading-into-writing construct in analytic rating scales may affect rating quality, and by extension score inferences and uses. Using two different analytic rating scales as proxies for two approaches to reading-into-writing construct operationalization, this study investigated the extent to which these approaches affect rating reliability and consistency. Twenty raters rated a set of reading-into-writing performances twice, each time using a different analytic rating scale, and completed post-rating questionnaires. The findings resulting from our convergent explanatory mixed-method research design show that both analytic rating scales functioned well, further supporting the use of analytic rating scales for scoring reading-into-writing. Raters reported that either type of analytic rating scale prompted them to attend to the reading-related aspects of reading-into-writing, although rating these aspects remained more challenging than judging writing-related aspects. The two scales differed, however, in the extent to which they led raters to uniform interpretations of performance difficulty levels. This study has implications for reading-into-writing scale design and rater training.

In this invited Viewpoint on the occasion of the 40th anniversary of the journal Language Testing, I argue that at the core of future challenges and opportunities for the field—both in scholarly and operational respects—remain basic questions and principles in language testing and assessment. Despite the high levels of sophistication of issues looked into, and methodological and operational solutions found, outstanding concerns still amount to: what are we testing, how are we testing, and why are we testing? Guided by these questions, I call for more thorough and adequate language use domain definitions (and a suitable broadening of research and testing methodologies to determine these), more comprehensive operationalizations of these domain definitions (especially in the context of technology in language testing), and deeper considerations of test purposes/uses and of their connections with domain definitions. To achieve this, I maintain that the field needs to continue investing in the topics of validation, ethics, and language assessment literacy, and engaging with broader fields of enquiry such as (applied) linguistics. I also encourage a more synthetic look at the existing knowledge base in order to build on this, and further diversification of voices in language testing and assessment research and practice.

Research has shown that language teachers typically feel underprepared for assessment aspects of their job. One reason may relate to how teacher education programmes prepare future teachers in this area. Research insights into how and to what extent teacher educators train future language teachers in language assessment matters are scarce, however, as are insights into the language assessment literacy (LAL) of the teacher educators themselves. Additionally, while increasingly research insights are available on components that constitute LAL, how such components interrelate is largely unexplored. To help address these research gaps, we investigated the LAL of English as a Foreign Language teacher educators in Chile. Through interviews with 20 teacher educators and analysis of their language assessment materials, five LAL components were identified (language assessment knowledge, conceptions, context, practices, and learning), and two by-products of LAL (language assessor identity and self-efficacy). The components were found to interrelate in a complex manner, which we visualized with a model of concentric oval shapes, depicting how LAL is socially constructed (and re-constructed) from and for the specific context in which teacher educators’ practices are immersed. We discuss implications for LAL conceptualisations and for LAL research methodology.

A long-standing debate in the testing of listening concerns the authenticity of the listening input. On the one hand, listening texts produced by item writers often lack spoken language characteristics. On the other hand, real-life recordings are often too context-specific to stand alone, or not suitable for item generation. In this study, we explored the effectiveness of an existing item-writing training course to produce authentic-sounding listening texts within the constraints of test specifications. Twenty-five trainees took an online item-writing course including training on creating authentic-sounding listening texts. Prior to and after the course, they developed a listening task. The resulting listening texts were judged on authenticity by three professional item reviewers and analysed linguistically by the researchers. Additionally, we interviewed the trainees following each item writing event and analysed their online discussions from during the course. Statistical comparison of the pre-and post-course authenticity scores revealed a positive effect of the training on item-writers’ ability to produce authentic-sounding listening texts, while the linguistic analysis demonstrated that the texts produced after the training contained more instances of spoken language. The interviews and discussions revealed that item writers’ awareness of spoken language features and their text production techniques influenced their ability to develop authentic-sounding texts.

Extensive research has demonstrated the impact of working memory (WM) on first language (L1) reading comprehension across age groups (Peng et al., 2018), and on foreign language (FL) reading comprehension of adults and older adolescents (Linck et al., 2014). Comparatively little is known about the effect of WM on young FL readers’ comprehension, and even less within testing contexts. Young FL readers are still developing their L1 reading skills and general cognitive skills (e.g., attentional regulation abilities). Completing FL reading tests might be particularly taxing on their WM, and differences in WM capacity – as well as other learner and task characteristics – might create construct-irrelevant variance in test performance. In this study, we investigate the effects of WM, grade level, and reading task on young learners’ FL reading test performances. Ninety-four young English language learners (Grades 6–7) in Hungary completed the TOEFL^® Junior™ Comprehensive’s reading test and a WM test battery. Our mixed-effects model predicted significantly higher comprehension accuracy among learners with higher WM capacity, and among learners in Grade 7 compared to learners in Grade 6. Reading task differences were not associated with significant comprehension accuracy differences. We discuss the implications of our findings for testing young learners’ FL reading comprehension.