Streamlined photoacoustic image processing with foundation models: A training-free solution
Open Access
- 27 August 2024
- journal article
- research article
- Published by World Scientific Pub Co Pte Ltd in Journal of Innovative Optical Health Sciences
- Vol. 18 (01)
Abstract
Foundation models (FMs) have rapidly evolved and have achieved significant accomplishments in computer vision tasks. Specifically, the prompt mechanism conveniently allows users to integrate image prior information into the model, making it possible to apply models without any training. Therefore, we proposed a workflow based on foundation models and zero training to solve the tasks of photoacoustic (PA) image processing. We employed the Segment Anything Model (SAM) by setting simple prompts and integrating the model’s outputs with prior knowledge of the imaged objects to accomplish various tasks, including: (1) removing the skin signal in three-dimensional PA image rendering; (2) dual speed-of-sound reconstruction, and (3) segmentation of finger blood vessels. Through these demonstrations, we have concluded that FMs can be directly applied in PA imaging without the requirement for network design and training. This potentially allows for a hands-on, convenient approach to achieving efficient and accurate segmentation of PA images. This paper serves as a comprehensive tutorial, facilitating the mastery of the technique through the provision of code and sample datasets.Keywords
All Related Versions
Funding Information
- Strategic Project of Precision Surgery, Tsinghua University
- Initiative Scientific Research Program, Institute for Intelligent Healthcare, Tsinghua University
- Tsinghua-Foshan Institute of Advanced Manufacturing
- National Natural Science Foundation of China (61735016)
- Beijing Nova Program (20230484308)
- Young Elite Scientists Sponsorship Program by CAST (2023QNRC001)
- Youth Elite Program of Beijing Friendship Hospital (YYQCJH2022-9)
- Science and Technology Program of Beijing Tongzhou District (KJ2023CX012)
This publication has 23 references indexed in Scilit:
- Review of large vision models and visual prompt engineeringMeta-Radiology, 2023
- A deep neural network for real-time optoacoustic image reconstruction with adjustable speed of soundNature Machine Intelligence, 2023
- Deep Learning Enhanced Volumetric Photoacoustic Imaging of Vasculature in HumanAdvanced Science, 2023
- Role of Chat GPT in Public HealthAnnals of Biomedical Engineering, 2023
- Full-view in vivo skin and blood vessels profile segmentation in photoacoustic imaging based on deep learningPhotoacoustics, 2021
- Massively parallel functional photoacoustic computed tomography of the human brainNature Biomedical Engineering, 2021
- Full three-dimensional segmentation and quantification of tumor vessels for photoacoustic imagesPhotoacoustics, 2020
- Automatic cancer tissue detection using multispectral photoacoustic imagingInternational Journal of Computer Assisted Radiology and Surgery, 2019
- Deep learning optoacoustic tomography with sparse dataNature Machine Intelligence, 2019
- Single-impulse panoramic photoacoustic computed tomography of small-animal whole-body dynamics at high spatiotemporal resolutionNature Biomedical Engineering, 2017