Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme
- 15 November 2008
- journal article
- research article
- Published by Taylor & Francis in Assessment & Evaluation in Higher Education
- Vol. 33 (6) , 699-709
Abstract
Traditionally the measures used to evaluate the impact of an educational programme on student outcomes and the extent to which students change is a comparison of the student's pre‐test scores with his/her post‐test scores. However, this method of evaluating change may be problematic due to the confounding factor of response shift bias when student self‐reports of change are used. Response shift bias occurs when the student's internal frame of reference of the construct being measured, for example research ability or critical thinking, changes between the pre‐test and the post‐test due to the influence of the educational programme. To control for response shift bias the retrospective pre‐test method was used to evaluate the outcomes achieved from students completing a research module at master's level. The retrospective pre‐test method differs from the traditional pre‐test–post‐test design in that both post‐test and pre‐test perceptions of respondents are collected at the same time. The findings indicated that response shift bias was evident in that the programme had significantly greater impact on outcomes than identified using the traditional pre‐test–post‐test design leading to the conclusion that students may overestimate their ability at the commencement of an educational programme. The retrospective pre‐test design is not a replacement for the traditional pre‐test–post‐test measures but may be a useful adjunct in the evaluation of the impact of educational programmes on student outcomes.Keywords
This publication has 25 references indexed in Scilit:
- Cognitive interviewing: verbal data in the design and pretesting of questionnairesJournal of Advanced Nursing, 2003
- Difficulties Experienced by Education and Sociology Students in Quantitative Methods CoursesStudies in Higher Education, 2003
- Self‐concept and Anxiety in University Students Studying Social Science Statistics Within a Co‐operative Learning StructureEducational Psychology, 1998
- The Role of Beliefs and Attitudes in Learning Statistics: Towards an Assessment FrameworkJournal of Statistics Education, 1994
- RESPONSE-SHIFT BIAS IN PROGRAM EVALUATIONImpact Assessment, 1989
- Pretesting effects in retrospective pretest-posttest designs.Journal of Applied Psychology, 1989
- On Delay and Reassessment of Retrospective PreratingsThe Journal of Experimental Education, 1988
- Using Retrospective Pre-Ratings to Counteract Response-Shift ConfoundingThe Journal of Experimental Education, 1982
- The Retrospective Pretest in an Educational Training ContextThe Journal of Experimental Education, 1982
- Response-shift bias: A source of contamination of self-report measures.Journal of Applied Psychology, 1979