Item Difficulty and Interviewer Knowledge Effects on the Accuracy and Consistency of Examinee Response Processes in Verbal Reports
- 9 April 2013
- journal article
- research article
- Published by Taylor & Francis in Applied Measurement in Education
- Vol. 26 (2) , 136-157
Abstract
The Standards for Educational and Psychological Testing indicate that multiple sources of validity evidence should be used to support the interpretation of test scores. In the past decade, examinee response processes, as a source of validity evidence, have received increased attention. However, there have been relatively few methodological studies of the accuracy and consistency of examinee response processes as measured by verbal reports in the context of educational measurement. The objective of the current study was to investigate the accuracy and consistency of examinee response processes—as measured by verbal reports—as a function of varying interviewer and item variables in a think aloud interview within an educational measurement context. Results indicate that the accuracy of responses may be undermined when students perceive the interviewer to be an expert in the domain. Further, the consistency of response processes may be undermined when items that are too easy or difficult are used to elicit reports. The implications of these results for conducting think-aloud studies are explored.Keywords
This publication has 32 references indexed in Scilit:
- Testing Expert-Based and Student-Based Cognitive Models: An Application of the Attribute Hierarchy Method and Hierarchy Consistency IndexApplied Measurement in Education, 2009
- Test Design with Cognition in MindEducational Measurement: Issues and Practice, 2006
- Avoiding Misconception, Misuse, and Missed Opportunities: The Collection of Verbal Reports in Educational Achievement TestingEducational Measurement: Issues and Practice, 2004
- Are We Asking the Right Questions? Using Cognitive Interviews to Improve Surveys in Education ResearchEducational Evaluation and Policy Analysis, 2004
- Does Teacher Certification Matter? High School Teacher Certification Status and Student AchievementEducational Evaluation and Policy Analysis, 2000
- Multiple goals, multiple pathways: The role of goal orientation in learning and achievement.Journal of Educational Psychology, 2000
- Effect of Eliciting Verbal Reports of Thinking on Critical Thinking Test PerformanceJournal of Educational Measurement, 1990
- Controlling for Background Beliefs When Developing Multiple‐Choice Critical Thinking TestsEducational Measurement: Issues and Practice, 1988
- Anxiety in verbal behavior: A validation study.Journal of Consulting Psychology, 1961
- Cue-counting: A measure of anxiety in interviews.Journal of Consulting Psychology, 1956