A New Method for Analyzing Content Validity Data Using Multidimensional Scaling
Authors:
- 25 January 2013
- journal article
- Published by SAGE Publications in Educational and Psychological Measurement
- Vol. 73 (3) , 365-385
Abstract
Validity evidence based on test content is of essential importance in educational testing. One source for such evidence is an alignment study, which helps evaluate the congruence between tested objectives and those specified in the curriculum. However, the results of an alignment study do not always sufficiently capture the degree to which a test adequately represents the intended content domain. In this study, we present and evaluate a method for analyzing alignment data that uses multidimensional scaling to determine whether the dimensions underlying subject matter experts’ (SME) ratings of test items conform to the dimensions delineated in the test specifications. The results suggest this procedure provides an enhanced view of content domain representation by portraying the content similarities among items along continuous, rather than discrete, dimensions. The additional types of information provided by this procedure, in relation to that provided by traditional analysis of SME data, are discussed.Keywords
This publication has 14 references indexed in Scilit:
- A Method to Examine Content Domain StructuresInternational Journal of Testing, 2011
- Evaluating Alignment Between Curriculum, Assessment, and InstructionReview of Educational Research, 2009
- Alignment of an Alternate Assessment with State Academic StandardsThe Journal of Special Education, 2005
- Benchmarking and Alignment of Standards and TestingEducational Assessment, 2004
- Gathering and Analyzing Content Validity DataEducational Assessment, 1998
- An Empirical Link of Content and Construct Validity EvidenceApplied Psychological Measurement, 1996
- Quantitative Methods for Assessing the Fit Between Test and CurriculumApplied Measurement in Education, 1989
- Content Validity and Reliability of Single Items or QuestionnairesEducational and Psychological Measurement, 1980
- A QUANTITATIVE APPROACH TO CONTENT VALIDITY1Personnel Psychology, 1975
- Analysis of Individual Differences in Multidimensional Scaling Via an N-way Generalization of “Eckart-Young” DecompositionPsychometrika, 1970