Nicole M. Duggan

Background: Point of care ultrasound (POCUS) is a critical skill for physicians across multiple medical specialties, yet substantial heterogeneity exists in how competency is assessed. Computer‐based approaches can be used to deliver, grade, and analyze learner performance, and may be more objective and reliable than traditional approaches using expert assessments. This study aimed to systematically review and summarize the existing literature surrounding computer‐based approaches to assessing POCUS competency.Methods: We searched six online databases (MEDLINE, IEEE Xplore Digital Library, Association for Computing Machinery Digital Library, PsycINFO (Ovid), EMBASE, Web of Science Core Collection). We included original peer‐reviewed studies that assessed computer‐based metrics of POCUS competence among any learner group performing POCUS. We also reviewed reference lists of all included studies. We extracted data elements that included the specialty of participants, POCUS experience, POCUS modality used, and type and results of computer‐based competency assessments. At least two authors conducted title and abstract screening, full text review, and data extraction, with discrepancies adjudicated by a third author. We present a qualitative synthesis of study findings.Results: Of 7375 identified studies, we included 28 in our final analysis. Computer‐based metrics were used to assess knowledge (n = 10), skills (n = 25), and cognitive load (n = 1) using hand tracking (n = 14), eye tracking (n = 7), image analysis (n = 6), and simulation scores (n = 1). In general, hand tracking analysis showed that experts had shorter probe path lengths, took less time to identify areas of interest, and had fewer discrete movements compared with novices. Eye tracking assessment showed increased dwell time was associated with successful completion of procedures and increased accuracy in interpreting images.Conclusion: We identified four computer‐based metrics for assessing POCUS competence, many of which demonstrated consistent performance in distinguishing skill level. Further work is needed to standardize and validate those approaches.

Background:. Currently, there is no standardization in ED-based-Ultrasound-guided-nerve-blocks (UGNB) training, credentialing, or procedural quality assurance nation-wide. This study aims to investigate objective measures of procedural competency for UGNB. Methods: Novice and expert users performed UGNB in a simulation-based setting. Participants were fitted with a head-motion tracking headband and their performance graded with a traditional OSCE. Head motion metrics were captured using both accelerometer and gyroscope sensors. Results: 11 novices and 7 experts were recruited. OSCE scores demonstrated a statistically significant difference between groups. Significant correlations were observed for accelerometer data in the z-axis and gyroscope data in the y-and z-axes, with positive correlation between accelerometer z-axis RMS values and OSCE scores(r=0.36), and gyroscope z-axis RMS values and OSCE scores(r=0.29). Conclusion: Objective measures of head motion in the z-axis demonstrated significant differences between novice and expert POCUS. Our data suggests that computer-based metrics may be reliable measures of procedural competency.

New sensors, technologies, data sources, and analytics have transformed the delivery of medical care in ambulances, emergency departments, observation units and, increasingly, in the burgeoning field of 'home hospital' treatment. Clinical decisions are increasingly based upon algorithms which may be derived from biased datasets. This minitrack focuses on research and practice at the intersection of technology and acute medical care, machine learning in emergency medical environments, and the healthcare impact of bias in algorithmic medicine.

Lung ultrasound (LUS) has become an indispensable tool at the bedside in emergency and acute care settings, offering a fast and non-invasive way to assess pulmonary congestion. Its portability and cost-effectiveness make it particularly valuable in resource-limited environments where quick decision-making is critical. Despite its advantages, the interpretation of B-line artifacts, which are key diagnostic indicators for conditions related to pulmonary congestion, can vary significantly among clinicians and even for the same clinician over time. This variability, coupled with the time pressure in acute settings, poses a challenge. To address this, our study introduces a new B-line segmentation method to calculate congestion scores from LUS images, aiming to standardize interpretations. We utilized a large dataset of 31,000 B-line annotations synthesized from over 550,000 crowdsourced opinions on LUS images of 299 patients to improve model training and accuracy. This approach has yielded a model with 94% accuracy in B-line counting (within a margin of 1) on a test set of 100 patients, demonstrating the potential of combining extensive data and crowdsourcing to refine lung ultrasound analysis for pulmonary congestion.

Background: Machine learning (ML) models can yield faster and more accurate medical diagnoses; however, developing ML models is limited by a lack of high-quality labeled training data. Crowdsourced labeling is a potential solution but can be constrained by concerns about label quality. Objective: This study aims to examine whether a gamified crowdsourcing platform with continuous performance assessment, user feedback, and performance-based incentives could produce expert-quality labels on medical imaging data. Methods: In this diagnostic comparison study, 2384 lung ultrasound clips were retrospectively collected from 203 emergency department patients. A total of 6 lung ultrasound experts classified 393 of these clips as having no B-lines, one or more discrete B-lines, or confluent B-lines to create 2 sets of reference standard data sets (195 training clips and 198 test clips). Sets were respectively used to (1) train users on a gamified crowdsourcing platform and (2) compare the concordance of the resulting crowd labels to the concordance of individual experts to reference standards. Crowd opinions were sourced from DiagnosUs (Centaur Labs) iOS app users over 8 days, filtered based on past performance, aggregated using majority rule, and analyzed for label concordance compared with a hold-out test set of expert-labeled clips. The primary outcome was comparing the labeling concordance of collated crowd opinions to trained experts in classifying B-lines on lung ultrasound clips. Results: Our clinical data set included patients with a mean age of 60.0 (SD 19.0) years; 105 (51.7%) patients were female and 114 (56.1%) patients were White. Over the 195 training clips, the expert-consensus label distribution was 114 (58%) no B-lines, 56 (29%) discrete B-lines, and 25 (13%) confluent B-lines. Over the 198 test clips, expert-consensus label distribution was 138 (70%) no B-lines, 36 (18%) discrete B-lines, and 24 (12%) confluent B-lines. In total, 99,238 opinions were collected from 426 unique users. On a test set of 198 clips, the mean labeling concordance of individual experts relative to the reference standard was 85.0% (SE 2.0), compared with 87.9% crowdsourced label concordance (P=.15). When individual experts’ opinions were compared with reference standard labels created by majority vote excluding their own opinion, crowd concordance was higher than the mean concordance of individual experts to reference standards (87.4% vs 80.8%, SE 1.6 for expert concordance; P<.001). Clips with discrete B-lines had the most disagreement from both the crowd consensus and individual experts with the expert consensus. Using randomly sampled subsets of crowd opinions, 7 quality-filtered opinions were sufficient to achieve near the maximum crowd concordance. Conclusions: Crowdsourced labels for B-line classification on lung ultrasound clips via a gamified approach achieved expert-level accuracy. This suggests a strategic role for gamified crowdsourcing in efficiently generating labeled image data sets for training ML systems.