A Method to Examine Content Domain Structures
Authors:
- 1 October 2011
- journal article
- research article
- Published by Taylor & Francis in International Journal of Testing
- Vol. 11 (4) , 295-307
Abstract
After a test is developed, most content validation analyses shift from ascertaining domain definition to studying domain representation and relevance because the domain is assumed to be set once a test exists. We present an approach that allows for the examination of alternative domain structures based on extant test items. In our example based on the 59 items from the 2004 Arizona state high school mathematics test, we asked 34 subject matter experts (SMEs) to judge the similarity of items based on any characteristics deemed important to them. After sorting items into categories, SME similarity data were scaled using multidimensional analysis. Scale values were then cluster analyzed to examine the domain structure derived from SME group responses. The SME similarity data yielded a cluster configuration that shared some commonality with the state model as stipulated in test specifications, but differed in critical ways. Confirmatory factor analysis of student item responses indicated that although both models fit the data the SME domain configuration produced better fit indices, suggesting that analyzing domain definition after construction can enhance the understanding of construct meaning.Keywords
This publication has 17 references indexed in Scilit:
- Evaluating Alignment Between Curriculum, Assessment, and InstructionReview of Educational Research, 2009
- Aligning Tests with States' Content Standards: Methods and IssuesEducational Measurement: Issues and Practice, 2003
- Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternativesStructural Equation Modeling: A Multidisciplinary Journal, 1999
- An Empirical Link of Content and Construct Validity EvidenceApplied Psychological Measurement, 1996
- A Comparison of Two Methods for Structuring Performance DomainsApplied Measurement in Education, 1992
- Quantitative Methods for Assessing the Fit Between Test and CurriculumApplied Measurement in Education, 1989
- Comparing partitionsJournal of Classification, 1985
- An Examination of Procedures for Determining the Number of Clusters in a Data SetPsychometrika, 1985
- Objective Criteria for the Evaluation of Clustering MethodsJournal of the American Statistical Association, 1971
- A Coefficient of Agreement for Nominal ScalesEducational and Psychological Measurement, 1960