Abstract
With the widespread interest and uptake of super-resolution ultrasound (SRUS) through localization and tracking of microbubbles, also known as ultrasound localization microscopy (ULM), many localization and tracking algorithms have been developed. ULM can image many centimeters into tissue in-vivo and track microvascular flow non-invasively with sub-diffraction resolution. In a significant community effort, we organized a challenge, Ultrasound Localization and TRacking Algorithms for Super-Resolution (ULTRA-SR). The aims of this paper are threefold: to describe the challenge organization, data generation, and winning algorithms; to present the metrics and methods for evaluating challenge entrants; and to report results and findings of the evaluation. Realistic ultrasound datasets containing microvascular flow for different clinical ultrasound frequencies were simulated, using vascular flow physics, acoustic field simulation and nonlinear bubble dynamics simulation. Based on these datasets, 38 submissions from 24 research groups were evaluated against ground truth using an evaluation framework with six metrics, three for localization and three for tracking. In-vivo mouse brain and human lymph node data were also provided, and performance assessed by an expert panel. Winning algorithms are described and discussed. The publicly available data with ground truth and the defined metrics for both localization and tracking present a valuable resource for researchers to benchmark algorithms and software, identify optimized methods/software for their data, and provide insight into the current limits of the field. In conclusion, Ultra-SR challenge has provided benchmarking data and tools as well as direct comparison and insights for a number of the state-of-the art localization and tracking algorithms.
Funding Information
  • U.K. Engineering and Physical Sciences Research Council (EP/T008970/1, EP/V04799X/1)
  • Chan Zuckerberg Foundation (2020-225443)
  • EPSRC Impact Acceleration Account Funding (EP/R511547/1)
  • Medical Research Council (MRC) Confidence in Concept Scheme at Imperial College (MC-PC-18050)
  • Chief Scientist Office Scotland (TCS/18/40)
  • Medical Research Council Confidence in Concept (MRC/CIC3/027)
  • Science and Technology Facilities Council Project (ST/M007804/1)
  • British Heart Foundation (FS/PhD/21/29113)
  • European Research Council (ERC) under the European Union Horizon H2020 Program/ERC Consolidator (772786-ResolveStroke)
  • Natural Sciences and Engineering Research Council of Canada
  • Deutsche Forschungsgemeinschaft (233312120, 445703531)
  • Canada Research Chair Program
  • Burroughs Wellcome Fund (1018212.03)
  • European Research Council (ERC), Synergy, through SUper Resolution ultrasound imaging using Erythrocytes (854796)
  • National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health (R21EB030072)
  • Beckman Institute Postdoctoral Fellowship
  • FV Hunt Postdoctoral Research Fellowship in Acoustics from the Acoustical Society of America
  • National Institute of Biomedical Imaging and Bioengineering of the National Institutes of Health (R21EB030072)

This publication has 54 references indexed in Scilit: