Andrea Gotzmann
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Andrea Gotzmann has a total of 11 co-authors. Here are the top co-authors by number of shared publications:
Debra Pugh
Medical Council of Canada, Ottawa, Canada
3 shared publicationsIlona Bartman
Medical Council of Canada, Ottawa, Canada
2 shared publicationsJohn Boulet
Medical Council of Canada, Ottawa, Canada
2 shared publicationsJudy McCormick
Medical Council of Canada, Ottawa, Canada
2 shared publicationsMathieu Wojcik
Medical Council of Canada, Ottawa, Canada
2 shared publicationsYichi Zhang
Medical Council of Canada, Ottawa, Canada
2 shared publications
Latest Publications
- Open Access
- 25 July 2024
- Published by Springer Nature in BMC Medical Education
- Vol. 24 (1) , 1-10
Conducting an objective structured clinical examination under COVID-restricted conditions
Background: The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. The purpose of this study was to gather evidence to support the reliability and validity of the modified NAC Examination. Methods: The modified NAC Examination was delivered to 2,433 candidates in 2020 and 2021. Cronbach’s alpha, decision consistency, and accuracy values were calculated. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs. female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs. non-CSA), postgraduate training (PGT) (no PGT vs. PGT), and language of examination (English vs. French). Criterion relationships were summarized using correlations within and between the NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results: Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were statistically significant. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions: The NAC Examination yields reliable total scores and pass/fail decisions. Expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that required modifications due to COVID.
- 25 July 2024
- Published by Springer Nature in BMC Medical Education
- Vol. 24 (1) , 1-10
Open AccessConducting an objective structured clinical examination under COVID-restricted conditions
Background: The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. The purpose of this study was to gather evidence to support the reliability and validity of the modified NAC Examination. Methods: The modified NAC Examination was delivered to 2,433 candidates in 2020 and 2021. Cronbach’s alpha, decision consistency, and accuracy values were calculated. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs. female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs. non-CSA), postgraduate training (PGT) (no PGT vs. PGT), and language of examination (English vs. French). Criterion relationships were summarized using correlations within and between the NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results: Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were statistically significant. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions: The NAC Examination yields reliable total scores and pass/fail decisions. Expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that required modifications due to COVID.
- Open Access
- 5 April 2024
- Published by Springer Nature in Research Square
Conducting an Objective Structured Clinical Examination under COVID-Restricted Conditions
Background The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. Reliability and validity evidence for the modified NAC Examination is presented. This paper outlines validity and reliability evidence for modifications required due to significant interruptions or changes to OSCEs. Methods Cronbach’s alpha, decision consistency, and accuracy values were calculated to quantify measurement error. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs non-CSA), postgraduate training (PGT) (no PGT vs PGT), and language of examination (English vs French). Criterion relationships were summarized using correlations within and between NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions The NAC Examination yields reliable total scores and pass/fail decisions, expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that had major modifications and interruptions.
- 5 April 2024
- Published by Springer Nature in Research Square
Open AccessConducting an Objective Structured Clinical Examination under COVID-Restricted Conditions
Background The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. Reliability and validity evidence for the modified NAC Examination is presented. This paper outlines validity and reliability evidence for modifications required due to significant interruptions or changes to OSCEs. Methods Cronbach’s alpha, decision consistency, and accuracy values were calculated to quantify measurement error. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs non-CSA), postgraduate training (PGT) (no PGT vs PGT), and language of examination (English vs French). Criterion relationships were summarized using correlations within and between NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions The NAC Examination yields reliable total scores and pass/fail decisions, expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that had major modifications and interruptions.
- 7 September 2016
- Published by Taylor & Francis in Teaching and Learning in Medicine
- Vol. 29 (1) , 52-58
Cheating in OSCEs: The Impact of Simulated Security Breaches on OSCE Performance
Construct: Valid score interpretation is important for constructs in performance assessments such as objective structured clinical examinations (OSCEs). An OSCE is a type of performance assessment in which a series of standardized patients interact with the student or candidate who is scored by either the standardized patient or a physician examiner. Background: In high-stakes examinations, test security is an important issue. Students accessing unauthorized test materials can create an unfair advantage and lead to examination scores that do not reflect students' true ability level. The purpose of this study was to assess the impact of various simulated security breaches on OSCE scores. Approach: Seventy-six 3rd-year medical students participated in an 8-station OSCE and were randomized to either a control group or to 1 of 2 experimental conditions simulating test security breaches: station topic (i.e., providing a list of station topics prior to the examination) or egregious security breach (i.e., providing detailed content information prior to the examination). Overall total scores were compared for the 3 groups using both a one-way between-subjects analysis of variance and a repeated measure analysis of variance to compare the checklist, rating scales, and oral question subscores across the three conditions. Results: Overall total scores were highest for the egregious security breach condition (81.8%), followed by the station topic condition (73.6%), and they were lowest for the control group (67.4%). This trend was also found with checklist subscores only (79.1%, 64.9%, and 60.3%, respectively for the security breach, station topic, and control conditions). Rating scale subscores were higher for both the station topic and egregious security breach conditions compared to the control group (82.6%, 83.1%, and 77.6%, respectively). Oral question subscores were significantly higher for the egregious security breach condition (88.8%) followed by the station topic condition (64.3%), and they were the lowest for the control group (48.6%). Conclusions: This simulation of different OSCE security breaches demonstrated that student performance is greatly advantaged by having prior access to test materials. This has important implications for medical educators as they develop policies and procedures regarding the safeguarding and reuse of test content.
- 7 September 2016
- Published by Taylor & Francis in Teaching and Learning in Medicine
- Vol. 29 (1) , 52-58
Cheating in OSCEs: The Impact of Simulated Security Breaches on OSCE Performance
Construct: Valid score interpretation is important for constructs in performance assessments such as objective structured clinical examinations (OSCEs). An OSCE is a type of performance assessment in which a series of standardized patients interact with the student or candidate who is scored by either the standardized patient or a physician examiner. Background: In high-stakes examinations, test security is an important issue. Students accessing unauthorized test materials can create an unfair advantage and lead to examination scores that do not reflect students' true ability level. The purpose of this study was to assess the impact of various simulated security breaches on OSCE scores. Approach: Seventy-six 3rd-year medical students participated in an 8-station OSCE and were randomized to either a control group or to 1 of 2 experimental conditions simulating test security breaches: station topic (i.e., providing a list of station topics prior to the examination) or egregious security breach (i.e., providing detailed content information prior to the examination). Overall total scores were compared for the 3 groups using both a one-way between-subjects analysis of variance and a repeated measure analysis of variance to compare the checklist, rating scales, and oral question subscores across the three conditions. Results: Overall total scores were highest for the egregious security breach condition (81.8%), followed by the station topic condition (73.6%), and they were lowest for the control group (67.4%). This trend was also found with checklist subscores only (79.1%, 64.9%, and 60.3%, respectively for the security breach, station topic, and control conditions). Rating scale subscores were higher for both the station topic and egregious security breach conditions compared to the control group (82.6%, 83.1%, and 77.6%, respectively). Oral question subscores were significantly higher for the egregious security breach condition (88.8%) followed by the station topic condition (64.3%), and they were the lowest for the control group (48.6%). Conclusions: This simulation of different OSCE security breaches demonstrated that student performance is greatly advantaged by having prior access to test materials. This has important implications for medical educators as they develop policies and procedures regarding the safeguarding and reuse of test content.