Isabelle Colmers-Gray
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Isabelle Colmers-Gray has a total of 62 co-authors. Here are the top co-authors by number of shared publications:
- Jeffrey A. Johnson
School of Public Health, University of Alberta, Edmonton, Canada
6 shared publications Sumit R. Majumdar
4 shared publicationsLeandro Solis Aguilar
Department of Biochemistry, Department of Medical Microbiology and Immunology, University of Alberta, Edmonton, Canada
3 shared publications- Lisa Tjosvold
University of Alberta
3 shared publications - Andrew Grock
Division of Emergency Medicine Greater Los Angeles VA Healthcare System Los Angeles California USA
2 shared publications Brandy A. Stauffer
Department of Emergency Medicine, University of Alberta, Edmonton, Canada
2 shared publications
Latest Publications
- 8 October 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- p. 1-4
Assessing health advocacy competencies through a novel simulation OSCE-based educational framework
Health advocacy remains one of the least explicitly assessed CanMEDS roles in Emergency Medicine. Objective structured clinical examination (OSCE) simulation is used to assess performance in real time but has not previously incorporated social determinants of health. We developed and piloted a simulation-based OSCE, embedding advocacy within acute clinical care and introduced novel assessment tools, completed by both examiners and standardized patients. Here we report the feasibility and acceptability of this OSCE and assessment tools. Twenty-two residents completed the advocacy and standard cases; 18 (82%) completed exit surveys. Most participants did not know advocacy would be assessed. Participants described the case as realistic and acceptable with minor suggestions for refinement. Assessors found the assessments intuitive to use and feasible to implement. This pilot suggests structured assessment of patient-level advocacy from both physician and patient perspectives can be feasibly integrated into simulation assessment without disrupting usual performance evaluation.
- 8 October 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- p. 1-4
Assessing health advocacy competencies through a novel simulation OSCE-based educational framework
Health advocacy remains one of the least explicitly assessed CanMEDS roles in Emergency Medicine. Objective structured clinical examination (OSCE) simulation is used to assess performance in real time but has not previously incorporated social determinants of health. We developed and piloted a simulation-based OSCE, embedding advocacy within acute clinical care and introduced novel assessment tools, completed by both examiners and standardized patients. Here we report the feasibility and acceptability of this OSCE and assessment tools. Twenty-two residents completed the advocacy and standard cases; 18 (82%) completed exit surveys. Most participants did not know advocacy would be assessed. Participants described the case as realistic and acceptable with minor suggestions for refinement. Assessors found the assessments intuitive to use and feasible to implement. This pilot suggests structured assessment of patient-level advocacy from both physician and patient perspectives can be feasibly integrated into simulation assessment without disrupting usual performance evaluation.
- 10 September 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (9) , 793-794
Scoring points—or missing the point? Rethinking pain assessment and analgesia in older adults
- 10 September 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (9) , 793-794
Scoring points—or missing the point? Rethinking pain assessment and analgesia in older adults
- 31 March 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (6) , 546-558
Teaching resuscitation of critically ill patients to medical learners in the clinical setting – A scoping review
Objectives Managing critically ill patients requires specific skills, yet little is known about how teaching occurs during real-time resuscitations. This scoping review aimed to summarize and map how teaching during real-time resuscitation has been described and conceptualized in the literature, to inform emergency physician clinician-educators and to identify gaps in this area. Methods We searched 6 databases from inception to April 2025 and hand-searched abstracts from 10 conferences held in the past 7 years. Backward and forward citation searches were performed. Eligible studies described a teaching interaction involving a medical learner (any level) and an actual critically ill patient in a hospital-based setting. No restrictions were applied based on publication type. Included eligible empirical studies were appraised using the Mixed Methods Assessment Tool. Using descriptive qualitative content analysis, we inductively and iteratively developed categories through team discussion to organize how teaching during resuscitation has been reported in the literature. Results A total of 25,530 titles and abstracts were screened. Full-text review of 242 articles yielded 13 included records, with 8 additional identified through reference screening, forward searching, and conference material review. Resuscitations most commonly occurred in the emergency department (n = 8) or the intensive care unit (n = 4). Learners were primarily residents (n = 17) or fellows (n = 7). Appraisal of 10 empirical studies showed heterogeneous quality. Twenty-three teaching strategies were described. Three categories emerged from the analysis: (1) teaching adapted to learners’ level of involvement during the resuscitation, (2) supervisor practices, and (3) ethical considerations shaping teaching during resuscitation. Conclusion This scoping review summarizes and maps how teaching during real-time resuscitation has been described and conceptualized in the literature. It highlights reported strategies that emergency physicians may consider adapting in clinical practice while also underscoring the paucity of empirical work in this underexplored area.
- 31 March 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (6) , 546-558
Teaching resuscitation of critically ill patients to medical learners in the clinical setting – A scoping review
Objectives Managing critically ill patients requires specific skills, yet little is known about how teaching occurs during real-time resuscitations. This scoping review aimed to summarize and map how teaching during real-time resuscitation has been described and conceptualized in the literature, to inform emergency physician clinician-educators and to identify gaps in this area. Methods We searched 6 databases from inception to April 2025 and hand-searched abstracts from 10 conferences held in the past 7 years. Backward and forward citation searches were performed. Eligible studies described a teaching interaction involving a medical learner (any level) and an actual critically ill patient in a hospital-based setting. No restrictions were applied based on publication type. Included eligible empirical studies were appraised using the Mixed Methods Assessment Tool. Using descriptive qualitative content analysis, we inductively and iteratively developed categories through team discussion to organize how teaching during resuscitation has been reported in the literature. Results A total of 25,530 titles and abstracts were screened. Full-text review of 242 articles yielded 13 included records, with 8 additional identified through reference screening, forward searching, and conference material review. Resuscitations most commonly occurred in the emergency department (n = 8) or the intensive care unit (n = 4). Learners were primarily residents (n = 17) or fellows (n = 7). Appraisal of 10 empirical studies showed heterogeneous quality. Twenty-three teaching strategies were described. Three categories emerged from the analysis: (1) teaching adapted to learners’ level of involvement during the resuscitation, (2) supervisor practices, and (3) ethical considerations shaping teaching during resuscitation. Conclusion This scoping review summarizes and maps how teaching during real-time resuscitation has been described and conceptualized in the literature. It highlights reported strategies that emergency physicians may consider adapting in clinical practice while also underscoring the paucity of empirical work in this underexplored area.
- 23 March 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (6) , 559-568
Are patients seen by a supervised learner more likely to return to the Emergency Department? A regional multi-centre cohort study of electronic health records
Introduction Teaching hospitals are vital for training future physicians, yet there is concern that the presence of learners may increase unscheduled emergency department (ED) return visits. Previous studies of this phenomenon are limited to a sample of EDs in a region. Our study aimed to address this gap using a comprehensive regional dataset and multiple academic years of routinely collected electronic medical record data. Our objective was to determine whether ED patients seen by supervised learners had higher rates of unscheduled return visits within 72 hours compared to those seen by attending physicians alone. Secondary outcomes were return visits resulting in hospital admission and the impact of learner training level. Methods We conducted a cohort study of over 1 million ED discharges across nine linked hospitals in a large Canadian health region between July 1, 2015, and June 30, 2018. The primary outcome was the rate of unscheduled return visits within 72 hours. Secondary outcomes included return visits with hospital admission and critical care admission. Logistic regression models were adjusted for patient demographics, acuity, and site characteristics. Results Among 1,033,026 patient visits, 7.4% returned within 72 hours. There was no statistically significant difference in unscheduled return visits between supervised learners and attending physicians alone (adjusted RR 0.98, 95% CI 0.96–1.00, p = 0.054). Return visit admissions were slightly higher among patients seen by learners (adjusted RR 1.06, 95% CI 1.01–1.13), primarily those seen by medical students and off-service residents. Emergency medicine residents in mid-training had lower return visit rates than attendings. Conclusions Patients seen by supervised learners in a regional ED network did not have increased rates of unscheduled return visits. Further research is warranted to explore factors underlying return visit admissions in learner-involved care.
- 23 March 2026
- Published by Springer Nature in Canadian Journal of Emergency Medicine
- Vol. 28 (6) , 559-568
Are patients seen by a supervised learner more likely to return to the Emergency Department? A regional multi-centre cohort study of electronic health records
Introduction Teaching hospitals are vital for training future physicians, yet there is concern that the presence of learners may increase unscheduled emergency department (ED) return visits. Previous studies of this phenomenon are limited to a sample of EDs in a region. Our study aimed to address this gap using a comprehensive regional dataset and multiple academic years of routinely collected electronic medical record data. Our objective was to determine whether ED patients seen by supervised learners had higher rates of unscheduled return visits within 72 hours compared to those seen by attending physicians alone. Secondary outcomes were return visits resulting in hospital admission and the impact of learner training level. Methods We conducted a cohort study of over 1 million ED discharges across nine linked hospitals in a large Canadian health region between July 1, 2015, and June 30, 2018. The primary outcome was the rate of unscheduled return visits within 72 hours. Secondary outcomes included return visits with hospital admission and critical care admission. Logistic regression models were adjusted for patient demographics, acuity, and site characteristics. Results Among 1,033,026 patient visits, 7.4% returned within 72 hours. There was no statistically significant difference in unscheduled return visits between supervised learners and attending physicians alone (adjusted RR 0.98, 95% CI 0.96–1.00, p = 0.054). Return visit admissions were slightly higher among patients seen by learners (adjusted RR 1.06, 95% CI 1.01–1.13), primarily those seen by medical students and off-service residents. Emergency medicine residents in mid-training had lower return visit rates than attendings. Conclusions Patients seen by supervised learners in a regional ED network did not have increased rates of unscheduled return visits. Further research is warranted to explore factors underlying return visit admissions in learner-involved care.
- Open Access
- 31 July 2024
- Published by Wiley in Academic Emergency Medicine
- Vol. 31 (12) , 1243-1255
Agreement between the Maslach Burnout Inventory and the Copenhagen Burnout Inventory among emergency physicians and trainees
Background: Emergency physicians have the highest rates of burnout among all specialties. Existing burnout tools include the Copenhagen Burnout Inventory (CBI) and single‐item measures from the Maslach Burnout Inventory (MBI). While both were designed to measure burnout, how they conceptualize this phenomenon differs and their agreement is unclear. Given the close conceptual relationship between emotional regulation strategies such as distancing and distraction with the MBI subscale of depersonalization, we examined agreement between the two inventories and association with emotional regulation strategies as a lens to explore the conceptualization of burnout.Methods: We conducted a cross‐sectional survey of adult and pediatric emergency physicians and trainees in Canada. Survey questions were pretested using written feedback and cognitive interviews. “Frequent use” of an emotional regulation strategy was “most” or “all” shifts (≥4 on 5‐point Likert scale). Burnout was defined as mean ≥50/100 on the CBI and scoring ≥5 (out of 7) on at least one of the single‐item measures from the MBI. Associations with burnout were examined using multivariable logistic regression.Results: Of 147 respondents, 44.2% were positive for burnout on the CBI and 44.9% on the single‐item measures from the MBI. Disagreement was 21.1% overall, ranging from 12.5% for older (≥55 years) physicians to 30.2% for younger (<35 years) physicians. Use of distraction and use of distancing were strongly associated with burnout on the single‐item measures (adjusted odds ratio [aOR] 14.4, 95% confidence interval [CI] 3.4–60.8]) and CBI (aOR 10.1, 95% CI 2.5–39.8, respectively.Conclusions: Despite near‐equal rates of burnout, agreement between the CBI and single‐item measures from the MBI varies and was lower for younger emergency physicians/trainees. While emotional regulation strategies were felt to be important in supporting a career in emergency medicine, they were strongly associated with burnout. Future research is needed to better understand this phenomenon and which tools to use to measure burnout.
- 31 July 2024
- Published by Wiley in Academic Emergency Medicine
- Vol. 31 (12) , 1243-1255
Open AccessAgreement between the Maslach Burnout Inventory and the Copenhagen Burnout Inventory among emergency physicians and trainees
Background: Emergency physicians have the highest rates of burnout among all specialties. Existing burnout tools include the Copenhagen Burnout Inventory (CBI) and single‐item measures from the Maslach Burnout Inventory (MBI). While both were designed to measure burnout, how they conceptualize this phenomenon differs and their agreement is unclear. Given the close conceptual relationship between emotional regulation strategies such as distancing and distraction with the MBI subscale of depersonalization, we examined agreement between the two inventories and association with emotional regulation strategies as a lens to explore the conceptualization of burnout.Methods: We conducted a cross‐sectional survey of adult and pediatric emergency physicians and trainees in Canada. Survey questions were pretested using written feedback and cognitive interviews. “Frequent use” of an emotional regulation strategy was “most” or “all” shifts (≥4 on 5‐point Likert scale). Burnout was defined as mean ≥50/100 on the CBI and scoring ≥5 (out of 7) on at least one of the single‐item measures from the MBI. Associations with burnout were examined using multivariable logistic regression.Results: Of 147 respondents, 44.2% were positive for burnout on the CBI and 44.9% on the single‐item measures from the MBI. Disagreement was 21.1% overall, ranging from 12.5% for older (≥55 years) physicians to 30.2% for younger (<35 years) physicians. Use of distraction and use of distancing were strongly associated with burnout on the single‐item measures (adjusted odds ratio [aOR] 14.4, 95% confidence interval [CI] 3.4–60.8]) and CBI (aOR 10.1, 95% CI 2.5–39.8, respectively.Conclusions: Despite near‐equal rates of burnout, agreement between the CBI and single‐item measures from the MBI varies and was lower for younger emergency physicians/trainees. While emotional regulation strategies were felt to be important in supporting a career in emergency medicine, they were strongly associated with burnout. Future research is needed to better understand this phenomenon and which tools to use to measure burnout.
- 9 June 2022
- Published by Wiley in AEM Education and Training
- Vol. 6 (3) , e10761
Keeping up with the literature: A current practice survey and qualitative needs assessment among emergency physicians
Purpose: Physicians face the challenge of staying current with a rapidly growing body of evidence and applying it to their practice. How emergency physicians (EPs) do so is unknown. The authors sought to describe and assess needs around EP patterns of evidence‐based medicine (EBM) and continuing medical education (CME) resource use.Methods: The authors conducted a multicenter, cross‐sectional study in 2019 across 12 tertiary care, community, and suburban emergency department (ED) sites in the greater area of Edmonton. Information on EBM/CME resource use along with barriers and facilitators to staying current was gathered using a rigorously developed survey tool, distributed electronically and by mail. Responses were tabulated and subgroups analyzed using MANOVA and ANOVA tests. Thematic analysis of comments used a phenomenological lens.Results: A total of 118 EPs (40.1%) completed the survey. Listening to podcasts, attending EM conferences, and subscription‐based resources were preferred for staying current. Resource use differed by years in practice but not by age, sex, training background, or site type. EBM had an important impact on respondents’ practice (average rating 3.8 out of 5, with 5 indicating “practice changing”). Time was an important barrier. Most (62.7%) felt that they did not spend enough time, despite spending a median of 4 to 5 h monthly on EBM. Facilitators (including journal club summaries or lists of practice‐relevant papers) had only moderate impacts. Thematic analysis found three themes (importance of EBM, implementation challenges, and dissemination of EBM) and 13 subthemes.Conclusion: EPs preferentially chose podcasts, conferences, and subscription‐based resources to stay current with EBM; time was the biggest barrier. These findings help ED leads and educators tailor CME to physician learning preferences to maximize application of EBM to clinical practice. The next steps include developing/curating resources and disseminating the survey on a larger scale to identify opportunities for shared virtual resources.
- 9 June 2022
- Published by Wiley in AEM Education and Training
- Vol. 6 (3) , e10761
Keeping up with the literature: A current practice survey and qualitative needs assessment among emergency physicians
Purpose: Physicians face the challenge of staying current with a rapidly growing body of evidence and applying it to their practice. How emergency physicians (EPs) do so is unknown. The authors sought to describe and assess needs around EP patterns of evidence‐based medicine (EBM) and continuing medical education (CME) resource use.Methods: The authors conducted a multicenter, cross‐sectional study in 2019 across 12 tertiary care, community, and suburban emergency department (ED) sites in the greater area of Edmonton. Information on EBM/CME resource use along with barriers and facilitators to staying current was gathered using a rigorously developed survey tool, distributed electronically and by mail. Responses were tabulated and subgroups analyzed using MANOVA and ANOVA tests. Thematic analysis of comments used a phenomenological lens.Results: A total of 118 EPs (40.1%) completed the survey. Listening to podcasts, attending EM conferences, and subscription‐based resources were preferred for staying current. Resource use differed by years in practice but not by age, sex, training background, or site type. EBM had an important impact on respondents’ practice (average rating 3.8 out of 5, with 5 indicating “practice changing”). Time was an important barrier. Most (62.7%) felt that they did not spend enough time, despite spending a median of 4 to 5 h monthly on EBM. Facilitators (including journal club summaries or lists of practice‐relevant papers) had only moderate impacts. Thematic analysis found three themes (importance of EBM, implementation challenges, and dissemination of EBM) and 13 subthemes.Conclusion: EPs preferentially chose podcasts, conferences, and subscription‐based resources to stay current with EBM; time was the biggest barrier. These findings help ED leads and educators tailor CME to physician learning preferences to maximize application of EBM to clinical practice. The next steps include developing/curating resources and disseminating the survey on a larger scale to identify opportunities for shared virtual resources.
- 1 April 2021
- Published by Wiley in AEM Education and Training
- Vol. 5 (3) , e10601
The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources
Background: Free Open‐Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.Objectives: We assessed the AAT’s usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.Methods: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.Results: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use—understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence‐based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT’s average measure ICC was 0.88 (95% CI = 0.77 to 0.95).Conclusions: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
- 1 April 2021
- Published by Wiley in AEM Education and Training
- Vol. 5 (3) , e10601
The revised Approved Instructional Resources score: An improved quality evaluation tool for online educational resources
Background: Free Open‐Access Medical education (FOAM) use among residents continues to rise. However, it often lacks quality assurance processes and residents receive little guidance on quality assessment. The Academic Life in Emergency Medicine Approved Instructional Resources tool (AAT) was created for FOAM appraisal by and for expert educators and has demonstrated validity in this context. It has yet to be evaluated in other populations.Objectives: We assessed the AAT’s usability in a diverse population of practicing emergency medicine (EM) physicians, residents, and medical students; solicited feedback; and developed a revised tool.Methods: As part of the Medical Education Translational Resources: Impact and Quality (METRIQ) study, we recruited medical students, EM residents, and EM attendings to evaluate five FOAM posts with the AAT and provide quantitative and qualitative feedback via an online survey. Two independent analysts performed a qualitative thematic analysis with discrepancies resolved through discussion and negotiated consensus. This analysis informed development of an initial revised AAT, which was then further refined after pilot testing among the author group. The final tool was reassessed for reliability.Results: Of 330 recruited international participants, 309 completed all ratings. The Best Evidence in Emergency Medicine (BEEM) score was the component most frequently reported as difficult to use. Several themes emerged from the qualitative analysis: for ease of use—understandable, logically structured, concise, and aligned with educational value. Limitations include deviation from questionnaire best practices, validity concerns, and challenges assessing evidence‐based medicine. Themes supporting its use include evaluative utility and usability. The author group pilot tested the initial revised AAT, revealing a total score average measure intraclass correlation coefficient (ICC) of moderate reliability (ICC = 0.68, 95% confidence interval [CI] = 0 to 0.962). The final AAT’s average measure ICC was 0.88 (95% CI = 0.77 to 0.95).Conclusions: We developed the final revised AAT from usability feedback. The new score has significantly increased usability, but will need to be reassessed for reliability in a broad population.
- 30 January 2018
- Published by Taylor & Francis in Teaching and Learning in Medicine
- Vol. 30 (3) , 294-302
Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study
Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources. Purpose: Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs. Approach: Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument. Results: Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25–0.56), ALiEM AIR (0.41, IQR 0.29–0.60) and METRIQ-8 (0.40, IQR 0.28–0.59). Conclusion: The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.
- 30 January 2018
- Published by Taylor & Francis in Teaching and Learning in Medicine
- Vol. 30 (3) , 294-302
Quality Evaluation Scores are no more Reliable than Gestalt in Evaluating the Quality of Emergency Medicine Blogs: A METRIQ Study
Construct: We investigated the quality of emergency medicine (EM) blogs as educational resources. Purpose: Online medical education resources such as blogs are increasingly used by EM trainees and clinicians. However, quality evaluations of these resources using gestalt are unreliable. We investigated the reliability of two previously derived quality evaluation instruments for blogs. Approach: Sixty English-language EM websites that published clinically oriented blog posts between January 1 and February 24, 2016, were identified. A random number generator selected 10 websites, and the 2 most recent clinically oriented blog posts from each site were evaluated using gestalt, the Academic Life in Emergency Medicine (ALiEM) Approved Instructional Resources (AIR) score, and the Medical Education Translational Resources: Impact and Quality (METRIQ-8) score, by a sample of medical students, EM residents, and EM attendings. Each rater evaluated all 20 blog posts with gestalt and 15 of the 20 blog posts with the ALiEM AIR and METRIQ-8 scores. Pearson's correlations were calculated between the average scores for each metric. Single-measure intraclass correlation coefficients (ICCs) evaluated the reliability of each instrument. Results: Our study included 121 medical students, 88 EM residents, and 100 EM attendings who completed ratings. The average gestalt rating of each blog post correlated strongly with the average scores for ALiEM AIR (r = .94) and METRIQ-8 (r = .91). Single-measure ICCs were fair for gestalt (0.37, IQR 0.25–0.56), ALiEM AIR (0.41, IQR 0.29–0.60) and METRIQ-8 (0.40, IQR 0.28–0.59). Conclusion: The average scores of each blog post correlated strongly with gestalt ratings. However, neither ALiEM AIR nor METRIQ-8 showed higher reliability than gestalt. Improved reliability may be possible through rater training and instrument refinement.