Andre F. DE Champlain
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Andre F. DE Champlain has a total of 13 co-authors. Here are the top co-authors by number of shared publications:
- Andre-Philippe Boulais
Research & Development, Medical Council of Canada, Ottawa, Ontario, Canada
1 shared publication Andrea Gotzmann
All authors are with Psychometrics and Assessment Services, Medical Council of Canada, Ottawa, Ontario. André F. De Champlain, PhD, is Director; Andrea Gotzmann, PhD, is Senior Research Psychometrician; and Sirius Qin, MS, is Statistical Analyst
1 shared publication- Andrew Dallas
Educational Research Methodology Department, School of Education, University of North Carolina at Greensboro, Greensboro, North Carolina, USA
1 shared publication - Christina St-Onge
Department of Medicine, Faculty of Medicine and Health Sciences Université de Sherbrooke Sherbrooke Quebec Canada
1 shared publication - Daniel J. Côté
Faculté de médecine et des sciences de la santé, Université de Sherbrooke, Sherbrooke, Canada
1 shared publication Daniel J. Klass
1 shared publication
Latest Publications
- 1 October 2016
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 8 (4) , 504-506
Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework
Article Citation: André F. De Champlain, Andrea Gotzmann, and Sirius Qin, (2016) Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework. Journal of Graduate Medical Education In-Press. doi: http://dx.doi.org/10.4300/JGME-D-15-00751.1
- 1 October 2016
- Published by Journal of Graduate Medical Education in Journal of Graduate Medical Education
- Vol. 8 (4) , 504-506
Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework
Article Citation: André F. De Champlain, Andrea Gotzmann, and Sirius Qin, (2016) Assessing the Reliability of Performance Assessment Scores: Some Considerations in Selecting an Appropriate Framework. Journal of Graduate Medical Education In-Press. doi: http://dx.doi.org/10.4300/JGME-D-15-00751.1
- Open Access
- 20 January 2016
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 13, 6
Calibrating the Medical Council of Canada’s Qualifying Examination Part I using an integrated item response theory framework: a comparison of models and designs
Purpose: The aim of this research was to compare different methods of calibrating multiple choice question (MCQ) and clinical decision making (CDM) components for the Medical Council of Canada’s Qualifying Examination Part I (MCCQEI) based on item response theory. Methods: Our data consisted of test results from 8,213 first time applicants to MCCQEI in spring and fall 2010 and 2011 test administrations. The data set contained several thousand multiple choice items and several hundred CDM cases. Four dichotomous calibrations were run using BILOG-MG 3.0. All 3 mixed item format (dichotomous MCQ responses and polytomous CDM case scores) calibrations were conducted using PARSCALE 4. Results: The 2-PL model had identical numbers of items with chi-square values at or below a Type I error rate of 0.01 (83/3,499 or 0.02). In all 3 polytomous models, whether the MCQs were either anchored or concurrently run with the CDM cases, results suggest very poor fit. All IRT abilities estimated from dichotomous calibration designs correlated very highly with each other. IRT-based pass-fail rates were extremely similar, not only across calibration designs and methods, but also with regard to the actual reported decision to candidates. The largest difference noted in pass rates was 4.78%, which occurred between the mixed format concurrent 2-PL graded response model (pass rate= 80.43%) and the dichotomous anchored 1-PL calibrations (pass rate= 85.21%). Conclusion: Simpler calibration designs with dichotomized items should be implemented. The dichotomous calibrations provided better fit of the item response matrix than more complex, polytomous calibrations.
- 20 January 2016
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 13, 6
Open AccessCalibrating the Medical Council of Canada’s Qualifying Examination Part I using an integrated item response theory framework: a comparison of models and designs
Purpose: The aim of this research was to compare different methods of calibrating multiple choice question (MCQ) and clinical decision making (CDM) components for the Medical Council of Canada’s Qualifying Examination Part I (MCCQEI) based on item response theory. Methods: Our data consisted of test results from 8,213 first time applicants to MCCQEI in spring and fall 2010 and 2011 test administrations. The data set contained several thousand multiple choice items and several hundred CDM cases. Four dichotomous calibrations were run using BILOG-MG 3.0. All 3 mixed item format (dichotomous MCQ responses and polytomous CDM case scores) calibrations were conducted using PARSCALE 4. Results: The 2-PL model had identical numbers of items with chi-square values at or below a Type I error rate of 0.01 (83/3,499 or 0.02). In all 3 polytomous models, whether the MCQs were either anchored or concurrently run with the CDM cases, results suggest very poor fit. All IRT abilities estimated from dichotomous calibration designs correlated very highly with each other. IRT-based pass-fail rates were extremely similar, not only across calibration designs and methods, but also with regard to the actual reported decision to candidates. The largest difference noted in pass rates was 4.78%, which occurred between the mixed format concurrent 2-PL graded response model (pass rate= 80.43%) and the dichotomous anchored 1-PL calibrations (pass rate= 85.21%). Conclusion: Simpler calibration designs with dichotomized items should be implemented. The dichotomous calibrations provided better fit of the item response matrix than more complex, polytomous calibrations.
- Open Access
- 15 April 2015
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 12, 11
Best-fit model of exploratory and confirmatory factor analysis of the 2010 Medical Council of Canada Qualifying Examination Part I clinical decision-making cases
Purpose: This study aims to assess the fit of a number of exploratory and confirmatory factor analysis models to the 2010 Medical Council of Canada Qualifying Examination Part I (MCCQE1) clinical decision-making (CDM) cases. The outcomes of this study have important implications for a range of domains, including scoring and test development. Methods: The examinees included all first-time Canadian medical graduates and international medical graduates who took the MCCQE1 in spring or fall 2010. The fit of one- to five-factor exploratory models was assessed for the item response matrix of the 2010 CDM cases. Five confirmatory factor analytic models were also examined with the same CDM response matrix. The structural equation modeling software program Mplus was used for all analyses. Results: Out of the five exploratory factor analytic models that were evaluated, a three-factor model provided the best fit. Factor 1 loaded on three medicine cases, two obstetrics and gynecology cases, and two orthopedic surgery cases. Factor 2 corresponded to pediatrics, and the third factor loaded on psychiatry cases. Among the five confirmatory factor analysis models examined in this study, three- and four-factor lifespan period models and the five-factor discipline models provided the best fit. Conclusion: The results suggest that knowledge of broad disciplinary domains best account for performance on CDM cases. In test development, particular effort should be placed on developing CDM cases according to broad discipline and patient age domains; CDM testlets should be assembled largely using the criteria of discipline and age.
- 15 April 2015
- Published by Korea Health Personnel Licensing Examination Institute in Journal of Educational Evaluation for Health Professions
- Vol. 12, 11
Open AccessBest-fit model of exploratory and confirmatory factor analysis of the 2010 Medical Council of Canada Qualifying Examination Part I clinical decision-making cases
Purpose: This study aims to assess the fit of a number of exploratory and confirmatory factor analysis models to the 2010 Medical Council of Canada Qualifying Examination Part I (MCCQE1) clinical decision-making (CDM) cases. The outcomes of this study have important implications for a range of domains, including scoring and test development. Methods: The examinees included all first-time Canadian medical graduates and international medical graduates who took the MCCQE1 in spring or fall 2010. The fit of one- to five-factor exploratory models was assessed for the item response matrix of the 2010 CDM cases. Five confirmatory factor analytic models were also examined with the same CDM response matrix. The structural equation modeling software program Mplus was used for all analyses. Results: Out of the five exploratory factor analytic models that were evaluated, a three-factor model provided the best fit. Factor 1 loaded on three medicine cases, two obstetrics and gynecology cases, and two orthopedic surgery cases. Factor 2 corresponded to pediatrics, and the third factor loaded on psychiatry cases. Among the five confirmatory factor analysis models examined in this study, three- and four-factor lifespan period models and the five-factor discipline models provided the best fit. Conclusion: The results suggest that knowledge of broad disciplinary domains best account for performance on CDM cases. In test development, particular effort should be placed on developing CDM cases according to broad discipline and patient age domains; CDM testlets should be assembled largely using the criteria of discipline and age.
- 28 August 2014
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 20 (3) , 581-594
Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments
Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada’s National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.
- 28 August 2014
- Published by Springer Nature in Advances in Health Sciences Education
- Vol. 20 (3) , 581-594
Examiners and content and site: Oh My! A national organization’s investigation of score variation in large-scale performance assessments
Examiner effects and content specificity are two well known sources of construct irrelevant variance that present great challenges in performance-based assessments. National medical organizations that are responsible for large-scale performance based assessments experience an additional challenge as they are responsible for administering qualification examinations to physician candidates at several locations and institutions. This study explores the impact of site location as a source of score variation in a large-scale national assessment used to measure the readiness of internationally educated physician candidates for residency programs. Data from the Medical Council of Canada’s National Assessment Collaboration were analyzed using Hierarchical Linear Modeling and Rasch Analyses. Consistent with previous research, problematic variance due to examiner effects and content specificity was found. Additionally, site location was also identified as a potential source of construct irrelevant variance in examination scores.
- Open Access
- 15 February 2014
- Published by Springer Nature in BMC Medical Education
- Vol. 14 (1) , 30
Multiple tutorial-based assessments: a generalizability study
Tutorial-based assessment commonly used in problem-based learning (PBL) is thought to provide information about students which is different from that gathered with traditional assessment strategies such as multiple-choice questions or short-answer questions. Although multiple-observations within units in an undergraduate medical education curriculum foster more reliable scores, that evaluation design is not always practically feasible. Thus, this study investigated the overall reliability of a tutorial-based program of assessment, namely the Tutotest-Lite.
- 15 February 2014
- Published by Springer Nature in BMC Medical Education
- Vol. 14 (1) , 30
Open AccessMultiple tutorial-based assessments: a generalizability study
Tutorial-based assessment commonly used in problem-based learning (PBL) is thought to provide information about students which is different from that gathered with traditional assessment strategies such as multiple-choice questions or short-answer questions. Although multiple-observations within units in an undergraduate medical education curriculum foster more reliable scores, that evaluation design is not always practically feasible. Thus, this study investigated the overall reliability of a tutorial-based program of assessment, namely the Tutotest-Lite.
- Open Access
- 1 October 1998
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 73 (10) , S114-116
SETTING STANDARDS FOR PERFORMANCE ASSESSMENT
An abstract is unavailable. This article is available as a PDF only.
- 1 October 1998
- Published by Oxford University Press (OUP) in Academic Medicine
- Vol. 73 (10) , S114-116
Open AccessSETTING STANDARDS FOR PERFORMANCE ASSESSMENT
An abstract is unavailable. This article is available as a PDF only.