LCLisa N. Conforti

Lisa N. Conforti

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Lisa N. Conforti has a total of 12 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    Orthopedic Surgery Milestones 2.0: Revisiting Program Director Opinion

    by Suzanne Elizabeth Ames, Lisa N. Conforti, Sonia Sangha, Tessa Balach, Kenji Yamazaki, Anuruk Lekdee, Laura Edgar

    • 17 February 2024
      • Vol. 16  (1) , 30-36

    Accuracy of Entrustment-Based Assessment: Implications for Programs and Patients.

    by C. Jessica Dine, Lisa N. Conforti, Eric S. Holmboe, Jennifer R. Kogan

    Background Although entrustment-supervision ratings are more intuitive compared to other rating scales, it is not known whether their use accurately assesses the appropriateness of care provided by a resident. Objective To determine the frequency of incorrect entrustment ratings assigned by faculty and whether accuracy of an entrustment-supervision scale differed by resident performance when the scripted resident performance level is known. Methods Faculty participants rated standardized residents in 10 videos using a 4-point entrustment-supervision scale. We calculated the frequency of rating a resident incorrectly. We performed generalizability (G) and decision (D) studies for all 10 cases (768 ratings) and repeated the analysis using only cases with an entrustment score of 2. Results The mean score by 77 raters for all videos was 2.87 (SD=0.86) with a mean of 2.37 (SD=0.72), 3.11 (SD=0.67) and 3.78 (SD=0.43) for the scripted levels of 2, 3, and 4. Faculty ratings differed from the scripted score for 331of 768 (43%) ratings. Most errors were ratings higher than the scripted score (223, 67%). G studies estimated the variance proportions of rater and case to be 4.99% and 54.29%. D studies estimated that 3 raters would need to watch 10 cases. The variance proportion of rater was 8.5% when the analysis was restricted to level 2 entrustment, requiring 15 raters to watch 5 cases. Conclusions Participants underestimated residents' potential need for greater supervision. Overall agreement between raters and scripted scores were low.


  • Gender Differences in Work-Based Assessment Scores and Narrative Comments After Direct Observation

    by Janae K. Heath, Jennifer R. Kogan, Eric S. Holmboe, Lisa Conforti, Yoon Soo Park, C. Jessica Dine

    Background While some prior studies of work-based assessment (WBA) numeric ratings have not shown gender differences, they have been unable to account for the true performance of the resident or explore narrative differences by gender. Objective To explore gender differences in WBA ratings as well as narrative comments (when scripted performance was known). Design Secondary analysis of WBAs obtained from a randomized controlled trial of a longitudinal rater training intervention in 2018–2019. Participating faculty (n = 77) observed standardized resident–patient encounters and subsequently completed rater assessment forms (RAFs). Subjects Participating faculty in longitudinal rater training. Main Measures Gender differences in mean entrustment ratings (4-point scale) were assessed with multivariable regression (adjusted for scripted performance, rater and resident demographics, and the interaction between study arm and time period [pre- versus post-intervention]). Using pre-specified natural language processing categories (masculine, feminine, agentic, and communal words), multivariable linear regression was used to determine associations of word use in the narrative comments with resident gender, race, and skill level, faculty demographics, and interaction between the study arm and the time period (pre- versus post-intervention). Key Results Across 1527 RAFs, there were significant differences in entrustment ratings between women and men standardized residents (2.29 versus 2.54, respectively, p < 0.001) after correction for resident skill level. As compared to men, feminine terms were more common for comments of what the resident did poorly among women residents (β 0.45, CI 0.12–0.78, p 0.01). This persisted despite adjusting for the faculty’s entrustment ratings. There were no other significant linguistic differences by gender. Conclusions Contrasting prior studies, we found entrustment rating differences in a simulated WBA which persisted after adjusting for the resident’s scripted performance. There were also linguistic differences by gender after adjusting for entrustment ratings, with feminine terms being used more frequently in comments about women in some, but not all narrative comments.


  • Open Access

    Faculty Perceptions of Frame of Reference Training to Improve Workplace-Based Assessment

    by Jennifer R. Kogan, Lisa N. Conforti, Eric S. Holmboe

    Background: Workplace-based assessment (WBA) is a key assessment strategy in competency-based medical education. However, its full potential has not been actualized secondary to concerns with reliability, validity, and accuracy. Frame of reference training (FORT), a rater training technique that helps assessors distinguish between learner performance levels, can improve the accuracy and reliability of WBA, but the effect size is variable. Understanding FORT benefits and challenges help improve this rater training technique. Objective: To explore faculty's perceptions of the benefits and challenges associated with FORT. Methods: Subjects were internal medicine and family medicine physicians (n=41) who participated in a rater training intervention in 2018 consisting of in-person FORT followed by asynchronous online spaced learning. We assessed participants' perceptions of FORT in post-workshop focus groups and an end-of-study survey. Focus groups and survey free text responses were coded using thematic analysis. Results: All subjects participated in 1 of 4 focus groups and completed the survey. Four benefits of FORT were identified: (1) opportunity to apply skills frameworks via deliberate practice; (2) demonstration of the importance of certain evidence-based clinical skills; (3) practice that improved the ability to discriminate between resident skill levels; and (4) highlighting the importance of direct observation and the dangers using proxy information in assessment. Challenges included time constraints and task repetitiveness. Conclusions: Participants believe that FORT training serves multiple purposes, including helping them distinguish between learner skill levels while demonstrating the impact of evidence-based clinical skills and the importance of direct observation.


  • Can Rater Training Improve the Quality and Accuracy of Workplace-Based Assessment Narrative Comments and Entrustment Ratings? A Randomized Controlled Trial

    by Jennifer R. Kogan, C. Jessica Md Dine, Lisa N. Conforti, Eric S. Holmboe

    Purpose: Prior research evaluating workplace-based assessment (WBA) rater training effectiveness has not measured improvement in narrative comment quality and accuracy, nor accuracy of prospective entrustment-supervision ratings. The purpose of this study was to determine whether rater training, using performance dimension and frame of reference training, could improve WBA narrative comment quality and accuracy. A secondary aim was to assess impact on entrustment rating accuracy. Method: This single blind, multi-institution, randomized controlled trial of a multifaceted, longitudinal rater training intervention consisted of in-person training followed by asynchronous online spaced learning. In 2018, investigators randomized 94 internal medicine and family medicine physicians involved with resident education. Participants assessed 10 scripted standardized resident-patient videos at baseline and follow-up. Differences in holistic assessment of narrative comment accuracy and specificity, accuracy of individual scenario observations, and entrustment rating accuracy were evaluated with t-tests. Linear regression assessed impact of participant demographics and baseline performance. Results: Seventy-seven participants completed the study. At follow-up, the intervention group (n = 41), compared to the control group (n = 36), had higher scores for narrative holistic specificity (2.76 vs 2.31, P < .001, Cohen V = .30), accuracy (2.37 vs 2.06, P < .001, Cohen V = .20) and mean quantity of accurate (6.14 vs 4.33, P < .001), inaccurate (3.53 vs 2.41, P < .001), and overall observations (2.61 vs 1.92, P = .002, Cohen V = .47). In aggregate, the intervention group had more accurate entrustment ratings (58.1% vs 49.7%, P = .006, Phi = .30). Baseline performance was significantly associated with performance on final assessments. Conclusions: Quality and specificity of narrative comments improved with rater training; the effect was mitigated by inappropriate stringency. Training improved accuracy of prospective entrustment-supervision ratings but the effect was more limited. Participants with lower baseline rating skill may benefit most from training.