Comparative Reliability of Structured Versus Unstructured Interviews in the Admission Process of a Residency Program
Authors:
- Danielle BlouinDBDanielle Blouin
- Andrew G. DayADAndrew G. Day
- Andrey PavlovAPAndrey Pavlov
- 1 December 2011
- journal article
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 3 (4) , 517-523
Abstract
Background: Although never directly compared, structured interviews are reported as being more reliable than unstructured interviews. This study compared the reliability of both types of interview when applied to a common pool of applicants for positions in an emergency medicine residency program. Methods: In 2008, one structured interview was added to the two unstructured interviews traditionally used in our resident selection process. A formal job analysis using the critical incident technique guided the development of the structured interview tool. This tool consisted of 7 scenarios assessing 4 of the domains deemed essential for success as a resident in this program. The traditional interview tool assessed 5 general criteria. In addition to these criteria, the unstructured panel members were asked to rate each candidate on the same 4 essential domains rated by the structured panel members. All 3 panels interviewed all candidates. Main outcomes were the overall, interitem, and interrater reliabilities, the correlations between interview panels, and the dimensionality of each interview tool. Results: Thirty candidates were interviewed. The overall reliability reached 0.43 for the structured interview, and 0.81 and 0.71 for the unstructured interviews. Analyses of the variance components showed a high interrater, low interitem reliability for the structured interview, and a high interrater, high interitem reliability for the unstructured interviews. The summary measures from the 2 unstructured interviews were significantly correlated, but neither was correlated with the structured interview. Only the structured interview was multidimensional. Conclusions: A structured interview did not yield a higher overall reliability than both unstructured interviews. The lower reliability is explained by a lower interitem reliability, which in turn is due to the multidimensionality of the interview tool. Both unstructured panels consistently rated a single dimension, even when prompted to assess the 4 specific domains established as essential to succeed in this residency program.Keywords
This publication has 26 references indexed in Scilit:
- Reliability of a Structured Interview for Admission to an Emergency Medicine Residency ProgramTeaching and Learning in Medicine, 2010
- Ophthalmology Resident Selection: Current Trends in Selection Criteria and Improving the ProcessOphthalmology, 2010
- Importance of the Faculty Interview During the Resident Application ProcessJournal of Surgical Education, 2007
- Resident Selection Process and Prediction of Clinical Performance in an Obstetrics and Gynecology ProgramTeaching and Learning in Medicine, 2006
- The Resident Application Process and Its Correlation to Future Performance as a ResidentAnesthesia & Analgesia, 2005
- A critique and standardization of meta‐analytic validity coefficients in personnel selectionJournal of Occupational and Organizational Psychology, 2001
- Performance Assessments from the Perspective of Generalizability TheoryApplied Psychological Measurement, 2000
- Methods of recruiting and selecting residents for U.S. family practice residenciesAcademic Medicine, 1994
- The Predictive Utility of Behavior-Based Interviewing Compared with Traditional Interviewing in the Selection of Radiology ResidentsInvestigative Radiology, 1992
- Criteria for Selection of Physical Medicine and Rehabilitation ResidentsAmerican Journal of Physical Medicine & Rehabilitation, 1989