Steven L. Wise
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Steven L. Wise has a total of 18 co-authors. Here are the top co-authors by number of shared publications:
Megan R. Kuhfeld
NWEA, Portland, Oregon, USA
3 shared publicationsJames Soland
NWEA, USA
2 shared publications- Ben Stenhaug
The Graduate School of Education at Stanford University, Stanford, CA, USA
1 shared publication - Benjamin W. Domingue
Graduate School of Education, Stanford University, Stanford, CA, USA
1 shared publication - Chris Piech
Department of Computer Science Stanford University
1 shared publication Dena A. Pastor
Center for Assessment and Research Studies , James Madison University
1 shared publication
Latest Publications
- 3 April 2022
- Published by Taylor & Francis in Educational Assessment
- Vol. 27 (2) , 136-151
Assessment in the Time of COVID-19: Understanding Patterns of Student Disengagement during Remote Low-Stakes Testing
The arrival of the COVID-19 pandemic had a profound effect on K-12 education. Most schools transitioned to remote instruction, and some used remote testing to assess student learning. Remote testing, however, is less controlled than in-school testing, leading to concerns regarding test-taking engagement. This study compared the disengagement of students remotely administered an adaptive interim assessment in spring 2020 with their disengagement on the assessment administered in-school during fall 2019. Results showed that disengagement gradually increased across grade level. This pattern was not meaningfully different between the two testing contexts, with the exception of results for American Indian/Alaska Native students, who showed higher disengagement under remote testing. In addition, the test’s engagement feature – which automatically paused the test event of a disengaged student and notified the test proctor – had a consistently positive impact whether the proctor was in the same room as the student or proctoring was done remotely.
- 3 April 2022
- Published by Taylor & Francis in Educational Assessment
- Vol. 27 (2) , 136-151
Assessment in the Time of COVID-19: Understanding Patterns of Student Disengagement during Remote Low-Stakes Testing
The arrival of the COVID-19 pandemic had a profound effect on K-12 education. Most schools transitioned to remote instruction, and some used remote testing to assess student learning. Remote testing, however, is less controlled than in-school testing, leading to concerns regarding test-taking engagement. This study compared the disengagement of students remotely administered an adaptive interim assessment in spring 2020 with their disengagement on the assessment administered in-school during fall 2019. Results showed that disengagement gradually increased across grade level. This pattern was not meaningfully different between the two testing contexts, with the exception of results for American Indian/Alaska Native students, who showed higher disengagement under remote testing. In addition, the test’s engagement feature – which automatically paused the test event of a disengaged student and notified the test proctor – had a consistently positive impact whether the proctor was in the same room as the student or proctoring was done remotely.
- 3 July 2021
- Published by Taylor & Francis in Educational Assessment
- Vol. 26 (3) , 163-174
The Impact of Disengaged Test Taking on a State’s Accountability Test Results
This study investigated test-taking engagement on the Spring 2019 administration of a large-scale state summative assessment. Through the identification of rapid-guessing behavior – which is a validated indicator of disengagement – the percentage of Grade 8 test events with meaningful amounts of rapid guessing was 5.5% in mathematics, 6.7% in English Language Arts (ELA), and 3.5% in science. Disengagement rates on the state summative test were also found to vary materially across gender, ethnicity, Individualized Educational Plan (IEP) status, Limited English Proficient (LEP) status, free and reduced lunch (FRL) status, and disability status. However, school mean performance, proficiency rates, and relative ranking were only minimally affected by disengagement. Overall, results of this study indicate that disengagement has a material impact on individual state summative test scores, though its impact on score aggregations may be relatively minor.
- 3 July 2021
- Published by Taylor & Francis in Educational Assessment
- Vol. 26 (3) , 163-174
The Impact of Disengaged Test Taking on a State’s Accountability Test Results
This study investigated test-taking engagement on the Spring 2019 administration of a large-scale state summative assessment. Through the identification of rapid-guessing behavior – which is a validated indicator of disengagement – the percentage of Grade 8 test events with meaningful amounts of rapid guessing was 5.5% in mathematics, 6.7% in English Language Arts (ELA), and 3.5% in science. Disengagement rates on the state summative test were also found to vary materially across gender, ethnicity, Individualized Educational Plan (IEP) status, Limited English Proficient (LEP) status, free and reduced lunch (FRL) status, and disability status. However, school mean performance, proficiency rates, and relative ranking were only minimally affected by disengagement. Overall, results of this study indicate that disengagement has a material impact on individual state summative test scores, though its impact on score aggregations may be relatively minor.
- 22 June 2021
- Published by Wiley in Journal of Educational Measurement
- Vol. 58 (3) , 335-363
Variation in Respondent Speed and its Implications: Evidence from an Adaptive Testing Scenario
The more frequent collection of response time data is leading to an increased need for an understanding of how such data can be included in measurement models. Models for response time have been advanced, but relatively limited large‐scale empirical investigations have been conducted. We take advantage of a large data set from the adaptive NWEA MAP Growth Reading Assessment to shed light on emergent features of response time behavior. We identify two behaviors in particular. The first, response acceleration, is a reduction in response time for responses that occur later in the assessment. We note that such reductions are heterogeneous as a function of estimated ability (lower ability estimates are associated with larger increases in acceleration) and that reductions in response time lead to lower accuracy relative to expectation for lower ability students. The second is within‐person variation in the association between time usage and accuracy. Idiosyncratic within‐person changes in response time have inconsistent implications for accuracy; in some cases additional response time predicts higher accuracy but in other cases additional response time predicts declines in accuracy. These findings have implications for models that incorporate response time and accuracy. Our approach may be useful in other studies of adaptive testing data.
- 22 June 2021
- Published by Wiley in Journal of Educational Measurement
- Vol. 58 (3) , 335-363
Variation in Respondent Speed and its Implications: Evidence from an Adaptive Testing Scenario
The more frequent collection of response time data is leading to an increased need for an understanding of how such data can be included in measurement models. Models for response time have been advanced, but relatively limited large‐scale empirical investigations have been conducted. We take advantage of a large data set from the adaptive NWEA MAP Growth Reading Assessment to shed light on emergent features of response time behavior. We identify two behaviors in particular. The first, response acceleration, is a reduction in response time for responses that occur later in the assessment. We note that such reductions are heterogeneous as a function of estimated ability (lower ability estimates are associated with larger increases in acceleration) and that reductions in response time lead to lower accuracy relative to expectation for lower ability students. The second is within‐person variation in the association between time usage and accuracy. Idiosyncratic within‐person changes in response time have inconsistent implications for accuracy; in some cases additional response time predicts higher accuracy but in other cases additional response time predicts declines in accuracy. These findings have implications for models that incorporate response time and accuracy. Our approach may be useful in other studies of adaptive testing data.
- 1 March 2021
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 34 (2) , 150-161
A Method for Identifying Partial Test-Taking Engagement
Effort-moderated (E-M) scoring is intended to estimate how well a disengaged test taker would have performed had they been fully engaged. It accomplishes this adjustment by excluding disengaged responses from scoring and estimating performance from the remaining responses. The scoring method, however, assumes that the remaining responses are not affected by disengagement. A recent study provided evidence that the responses used in E-M scoring can sometimes reflect less-than-full, or partial engagement. In these instances, E-M scores will be less trustworthy because they can still be distorted by disengagement. This paper describes a method for identifying partial engagement and provides validation evidence to support its use and interpretation. When test events indicate the presence of partial engagement, effort-moderated scores should be interpreted cautiously.
- 1 March 2021
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 34 (2) , 150-161
A Method for Identifying Partial Test-Taking Engagement
Effort-moderated (E-M) scoring is intended to estimate how well a disengaged test taker would have performed had they been fully engaged. It accomplishes this adjustment by excluding disengaged responses from scoring and estimating performance from the remaining responses. The scoring method, however, assumes that the remaining responses are not affected by disengagement. A recent study provided evidence that the responses used in E-M scoring can sometimes reflect less-than-full, or partial engagement. In these instances, E-M scores will be less trustworthy because they can still be distorted by disengagement. This paper describes a method for identifying partial engagement and provides validation evidence to support its use and interpretation. When test events indicate the presence of partial engagement, effort-moderated scores should be interpreted cautiously.
- 11 August 2020
- Published by Taylor & Francis in Educational Research and Evaluation
- Vol. 26 (5-6) , 328-338
Six insights regarding test-taking disengagement
There has been increasing concern about the presence of disengaged test taking in international assessment programmes and its implications for the validity of inferences made regarding a country’s level of educational attainment. This issue has received a growing research interest over the past 20 years, with notable advances in both the measurement of disengagement as well as our understanding of its distortive impact on both individual and aggregated scores. In this paper, I discuss six important insights yielded by this research and their implications for assessment programmes.
- 11 August 2020
- Published by Taylor & Francis in Educational Research and Evaluation
- Vol. 26 (5-6) , 328-338
Six insights regarding test-taking disengagement
There has been increasing concern about the presence of disengaged test taking in international assessment programmes and its implications for the validity of inferences made regarding a country’s level of educational attainment. This issue has received a growing research interest over the past 20 years, with notable advances in both the measurement of disengagement as well as our understanding of its distortive impact on both individual and aggregated scores. In this paper, I discuss six important insights yielded by this research and their implications for assessment programmes.
- 1 July 2020
- Published by Wiley in Journal of Educational Measurement
- Vol. 58 (1) , 130-149
Using Retest Data to Evaluate and Improve Effort‐Moderated Scoring
There has been a growing research interest in the identification and management of disengaged test taking, which poses a validity threat that is particularly prevalent with low‐stakes tests. This study investigated effort‐moderated (E‐M) scoring, in which item responses classified as rapid guesses are identified and excluded from scoring. Using achievement test data composed of test takers who were quickly retested and showed differential degrees of disengagement, three basic findings emerged. First, standard E‐M scoring accounted for roughly one‐third of the score distortion due to differential disengagement. Second, a modified E‐M scoring method that used more liberal time thresholds performed better—accounting for two‐thirds or more of the distortion. Finally, the inability of E‐M scoring to account for all of the score distortion suggests the additional presence of nonrapid item responses that reflect less‐than‐full engagement by some test takers.
- 1 July 2020
- Published by Wiley in Journal of Educational Measurement
- Vol. 58 (1) , 130-149
Using Retest Data to Evaluate and Improve Effort‐Moderated Scoring
There has been a growing research interest in the identification and management of disengaged test taking, which poses a validity threat that is particularly prevalent with low‐stakes tests. This study investigated effort‐moderated (E‐M) scoring, in which item responses classified as rapid guesses are identified and excluded from scoring. Using achievement test data composed of test takers who were quickly retested and showed differential degrees of disengagement, three basic findings emerged. First, standard E‐M scoring accounted for roughly one‐third of the score distortion due to differential disengagement. Second, a modified E‐M scoring method that used more liberal time thresholds performed better—accounting for two‐thirds or more of the distortion. Finally, the inability of E‐M scoring to account for all of the score distortion suggests the additional presence of nonrapid item responses that reflect less‐than‐full engagement by some test takers.
- 3 March 2020
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 33 (2) , 83-94
The Impact of Test-Taking Disengagement on Item Content Representation
In achievement testing there is typically a practical requirement that the set of items administered should be representative of some target content domain. This is accomplished by establishing test blueprints specifying the content constraints to be followed when selecting the items for a test. Sometimes, however, students give disengaged responses to some of their test items, which raises the issue of the degree to which the set of engaged responses maintain the intended content representation. The current investigation reports the results of two studies focused on rapid-guessing behavior. The first study showed evidence that differential rapid guessing often resulted in test events with meaningfully distorted content representation. The second study found that the differences in test taking engagement across content categories were primarily due to differences in the reading load of items. Implications for test-score validity are discussed along with suggestions for addressing the problem.
- 3 March 2020
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 33 (2) , 83-94
The Impact of Test-Taking Disengagement on Item Content Representation
In achievement testing there is typically a practical requirement that the set of items administered should be representative of some target content domain. This is accomplished by establishing test blueprints specifying the content constraints to be followed when selecting the items for a test. Sometimes, however, students give disengaged responses to some of their test items, which raises the issue of the degree to which the set of engaged responses maintain the intended content representation. The current investigation reports the results of two studies focused on rapid-guessing behavior. The first study showed evidence that differential rapid guessing often resulted in test events with meaningfully distorted content representation. The second study found that the differences in test taking engagement across content categories were primarily due to differences in the reading load of items. Implications for test-score validity are discussed along with suggestions for addressing the problem.
- Open Access
- 31 January 2020
- Published by International Association for Computerized Adaptive Testing in Journal of Computerized Adaptive Testing
- Vol. 8 (1) , 1-19
Three Measures of Test Adaptation Based on Optimal Test Information
Three Measures of Test Adaptation Based on Optimal Test Information
- 31 January 2020
- Published by International Association for Computerized Adaptive Testing in Journal of Computerized Adaptive Testing
- Vol. 8 (1) , 1-19
Open AccessThree Measures of Test Adaptation Based on Optimal Test Information
Three Measures of Test Adaptation Based on Optimal Test Information
- 26 September 2019
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 32 (4) , 325-336
An Information-Based Approach to Identifying Rapid-Guessing Thresholds
The identification of rapid guessing is important to promote the validity of achievement test scores, particularly with low-stakes tests. Effective methods for identifying rapid guesses require reliable threshold methods that are also aligned with test taker behavior. Although several common threshold methods are based on rapid guessing response accuracy or visual inspection of response time distributions, this paper describes a new information-based approach to setting thresholds that does not share the limitations of other methods. A pair of information-based methods are introduced, and an empirical comparison study found the new methods to more reliably set thresholds than methods based on response accuracy or visual inspection.
- 26 September 2019
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 32 (4) , 325-336
An Information-Based Approach to Identifying Rapid-Guessing Thresholds
The identification of rapid guessing is important to promote the validity of achievement test scores, particularly with low-stakes tests. Effective methods for identifying rapid guesses require reliable threshold methods that are also aligned with test taker behavior. Although several common threshold methods are based on rapid guessing response accuracy or visual inspection of response time distributions, this paper describes a new information-based approach to setting thresholds that does not share the limitations of other methods. A pair of information-based methods are introduced, and an empirical comparison study found the new methods to more reliably set thresholds than methods based on response accuracy or visual inspection.
- 15 May 2019
- Published by Taylor & Francis in International Journal of Testing
- Vol. 20 (1) , 57-77
The (Non)Impact of Differential Test Taker Engagement on Aggregated Scores
Disengaged test taking tends to be most prevalent with low-stakes tests. This has led to questions about the validity of aggregated scores from large-scale international assessments such as PISA and TIMSS, as previous research has found a meaningful correlation between the mean engagement and mean performance of countries. The current study, using data from the computer-based version of the PISA-Based Test for Schools, examined the distortive effects of differential engagement on aggregated school-level scores. The results showed that, although there was considerable differential engagement among schools, the school means were highly stable due to two factors. First, any distortive effects of disengagement in a school were diluted by a high proportion of the students exhibiting no non-effortful behavior. Second, and most interestingly, disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on the school’s mean.
- 15 May 2019
- Published by Taylor & Francis in International Journal of Testing
- Vol. 20 (1) , 57-77
The (Non)Impact of Differential Test Taker Engagement on Aggregated Scores
Disengaged test taking tends to be most prevalent with low-stakes tests. This has led to questions about the validity of aggregated scores from large-scale international assessments such as PISA and TIMSS, as previous research has found a meaningful correlation between the mean engagement and mean performance of countries. The current study, using data from the computer-based version of the PISA-Based Test for Schools, examined the distortive effects of differential engagement on aggregated school-level scores. The results showed that, although there was considerable differential engagement among schools, the school means were highly stable due to two factors. First, any distortive effects of disengagement in a school were diluted by a high proportion of the students exhibiting no non-effortful behavior. Second, and most interestingly, disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on the school’s mean.