Hatice Cigdem Bulut

Low-stakes assessments in K-12 education play a crucial role in monitoring student progress, yet their validity is often compromised by disengaged test-taking behaviors, such as rapid guessing and idling. This study introduces a novel application of the Nested Logit Model (NLM) to integrate test-taking engagement, as indicated by response times (RTs), with item responses to improve ability estimation. Using data from a low-stakes reading assessment in the United States (n = 27,556 students in grades 5 to 8), we compared six scoring approaches, including traditional dichotomous scoring, effort-moderated scoring, and nominal scoring that categorized responses based on RT-informed engagement. Our results demonstrated that nominal scoring approaches, particularly those distinguishing rapid guesses and idle responses, yielded superior model fit, increased measurement precision, and provided nuanced insights into examinee behaviors compared to dichotomous scoring methods. Latent class analysis further identified three distinct engagement profiles—effortful responders, rapid guessers, and idle responders—highlighting the need to address both rapid and idle behaviors in modeling. This study emphasizes the value of leveraging RT data to enhance the accuracy of low-stakes assessments while preserving response information. Findings also suggest the NLM framework as a practical and accessible tool for researchers and practitioners seeking to address disengaged behaviors and ensure the reliability of low-stakes assessments.

Mental imagery is a vital cognitive skill that significantly influences how reality is perceived while creating art. Its multifaceted nature reveals various dimensions of creative expression, amplifying the inherent complexities of measuring it. This study aimed to shorten the Mental Imagery Scale in Artistic Creativity (MISAC) via the Ant Colony Optimization algorithm (ACO), a metaheuristic methodology for developing psychometrically robust brief scales. Answering 63 items in the original version of MISAC demands a higher cognitive load and, consequently, more time. Therefore, our goal was to shorten it while preserving its psychometric properties. In this study, responses to the MISAC were obtained from 500 undergraduate students enrolled in an art education program. The items on the short form of the MISAC were selected based on pre-specified validity criteria and content representability. The 28-item short form of MISAC demonstrated comparable performance to the original version regarding construct validity, criteria-related validity, and reliability coefficients. Moreover, strict invariance was attained across both gender groups in the validation process of the short form. These results highlight the utility of the shortened version of the MISAC as a valid measure with minimal loss of information of scores compared to the full version.

The increasing volume of large-scale assessment data poses a challenge for testing organizations to manage data and conduct psychometric analysis efficiently. Traditional psychometric software presents barriers, such as a lack of functionality for managing data and conducting various standard psychometric analyses efficiently. These challenges have resulted in high costs to achieve the desired research and analysis outcomes. To address these challenges, we have designed and implemented a modernized data pipeline that allows psychometricians and statisticians to efficiently manage the data, conduct psychometric analysis, generate technical reports, and perform quality assurance to validate the required outputs. This modernized pipeline has proven to scale with large databases, decrease human error by reducing manual processes, efficiently make complex workloads repeatable, ensure high quality of the outputs, and reduce overall costs of psychometric analysis of large-scale assessment data. This paper aims to provide information to support the modernization of the current psychometric analysis practices. We shared details on the workflow design and functionalities of our modernized data pipeline, which provide a universal interface to large-scale assessments. The methods for developing non-technical and user-friendly interfaces will also be discussed.

Some statewide testing programs allow students to receive corrective feedback and revise their answers during testing. Despite its pedagogical benefits, the effects of providing revision opportunities remain unknown in the context of alternate assessments. Therefore, this study examined student data from a large-scale alternate assessment that allows students to make multiple attempts until they find the correct answer to multiple-choice items. The students receive partial credit based on the number of attempts being made. The effects of the multiple-attempt approach on both test characteristics and student performance were investigated. The results indicated that, despite making most items on the assessment relatively easier, the availability of partial credit improved the strength of the items in distinguishing low-achieving and high-achieving students while maintaining high internal consistency among the test items. Although the students were able to increase their scores due to the inclusion of partial credit based on the number of attempts, the relative positions of the students remained nearly the same.

This study examined group differences in online reading comprehension (ORC) using student data from the 2016 administration of the Progress in International Reading Literacy Study (ePIRLS). An explanatory item response modeling approach was used to explore the effects of item properties (i.e., item format, text complexity, and cognitive complexity), student characteristics (i.e., gender and language groups), and their interactions on dichotomous and polytomous item responses. The results showed that female students outperform male students in ORC tasks and that the achievement difference between female and male students appears to change text complexity increases. Similarly, the cognitive complexity of the items seems to play a significant role in explaining the gender gap in ORC performance. Students who never (or sometimes) speak the test language at home particularly struggled with answering ORC tasks. The achievement gap between students who always (or almost always) speak the test language at home and those who never (or sometimes) speak the test language at home was larger for constructed-response items and items with higher cognitive complexity. Overall, the findings suggest that item properties could help understand performance differences between gender and language groups in ORC assessments.

While the inclusion of both positively and negatively worded items is a common practice in scales, using positively and negatively worded items together may threaten the validity of a scale. Several studies have been devoted to investigating the effects of item wording methods. The current study investigated item wording effects on the responses of 4028 Turkish fifth-grade students, who responded to the Students Confidence in Mathematics (SCM) and Students Confidence in Science (SCS) scales. The role of early literacy-related variables (i.e., early literacy activities undertaken before primary school, student performance on reading literacy tasks upon entering primary school, and duration of the children’s pre-primary school attendance) on item wording effects was also examined. The investigations were conducted using confirmatory factor analysis and the correlated trait–correlated method minus one CFA- CTC(M-1) model, derived from the correlated traits-correlated methods framework. The results indicate that significant item wording effects existed in both scales. Moreover, a significant and positive effect was found in both scales relating to early literacy activities undertaken before school, but no effects were found relating to student performance on reading literacy tasks upon entering primary school or duration of the children’s pre-primary school attendance. Overall, the study suggests that researchers and practitioners should consider potential effects when including both positively and negatively worded items in scales, especially scales designed for younger students.

Several studies have been published on disengaged test respondents, and others have analyzed disengaged survey respondents separately. For many large-scale assessments, students answer questionnaire and test items in succession. This study examines the percentage of students who continuously engage in disengaged responding behaviors across sections in a low-stakes assessment. The effects on calculated scores of filtering students, based on their responding behaviors, are also analyzed. Data of this study came from the 2015 administration of PISA. For data analysis, frequencies and percentages of engaged students in the sessions were initially calculated using students' response times. To investigate the impact of filtering disengaged respondents on parameter estimation, three groups were created, namely engaged in both measures, engaged only in the test, and engaged only in the questionnaire. Next, several validity checks were performed on each group to verify the accuracy of the classifications and the impact of filtering student groups based on their responding behavior. The results indicate that students who are disengaged in tests tend to continue this behavior when responding to the questionnaire items in PISA. Moreover, the rate of continuity of disengaged responding is non-negligible as can be seen from the effect sizes. On the other hand, removing disengaged students in both measures led to higher or nearly the same performance ratings compared to the other groups. Researchers analyzing the dataset including achievement tests and survey items are recommended to review disengaged responses and filter out students who are continuously showing disengaged responding before performing further statistical analysis.

The Woodcock-Johnson IV Tests of Cognitive Abilities (WJ IV COG) is a comprehensive assessment battery designed to assess broad and narrow cognitive abilities, as defined by the Cattell-Horn-Carroll (CHC) theory of intelligence. Previous studies examined the invariance of the WJ assessments across sex and age groups using factor analytic methods. Psychometric network modeling is an alternative methodology that can address both direct and indirect relationships among the observed variables. In this study, we employed psychometric network modeling to examine the invariance of the WJ IV COG across sex and age groups. Using a normative sample (n = 4212 participants) representative of the United States population, we tested the extent to which the factorial structure of the WJ IV COG aligned with CHC theory for the school-aged sample. Next, we used psychometric network modeling as a data-driven method to investigate whether the network structure of the WJ IV COG remains similar across different sex and age (age 6 to 19, inclusively) groups. Our results showed that the WJ IV COG maintained the same network structure across all age and sex groups, although the network structure at younger ages indicated weaker relationships among some subtests. Overall, the results provide construct validity evidence for the WJ IV COG, based on both theoretical and data-driven methods.