Dimensionality and Generalizability of Domain-Independent Performance Assessments
Authors:
- 1 March 1996
- journal article
- research article
- Published by Taylor & Francis in The Journal of Educational Research
- Vol. 89 (4) , 197-205
Abstract
Empirical guidance for the design of comparable performance assessments is sorely lacking. A study was conducted to assess the degree to which domain specifications control topic and rater variability, focusing on task generalizability, rater reliability, and scoring rubric dimensionality. Two classes of history students were administered three on-demand, multistep performance tasks a week apart. For each topic, all students completed a Prior Knowledge Test, read primary source materials, and wrote an essay of explanation. Using a theory-based scoring rubric, four trained raters scored all essays. Inter- and intrarater reliabilities and g-study results are reported. Results show relative efficiency for the assessment approach. The dimensionality analysis supported two factors: Deep Understanding and Surface Understanding across the three topics. Prior Knowledge scores and GPA in history courses correlated with the Deep Understanding elements of the scoring rubric. Implications for design and testing purposes are discussed.Keywords
This publication has 8 references indexed in Scilit:
- Educational Assessment: Expanded Expectations and ChallengesEducational Evaluation and Policy Analysis, 1993
- Shifting Conceptions of Validity in Educational Measurement: Implications for Performance AssessmentReview of Educational Research, 1992
- Evaluation of Procedure‐Based Scoring for Hands‐On Science AssessmentJournal of Educational Measurement, 1992
- Using Performance Assessment for Accountability PurposesEducational Measurement: Issues and Practice, 1992
- Hypertext as a strategy for teaching and assessing knowledge representation1992
- Complex, Performance-Based Assessment: Expectations and Validation CriteriaEducational Researcher, 1991
- Psychometricians’ Beliefs About LearningEducational Researcher, 1991
- A Systems Approach to Educational TestingEducational Researcher, 1989