Raptor: Scalable Train-Free Embeddings for 3D Medical Volumes Leveraging Pretrained 2D Foundation Models
Authors:
- Ulzee AnUAUlzee An
- Moonseong JeongMJMoonseong Jeong
- Simon A. LeeSLSimon A. Lee
- Aditya GorlaAGAditya Gorla
- Yuzhe YangYYYuzhe Yang
- Sriram SankararamanSSSriram Sankararaman
Preprint
Abstract
Current challenges in developing foundational models for volumetric imaging data, such as magnetic resonance imaging (MRI), stem from the computational complexity of training state-of-the-art architectures in high dimensions and curating sufficiently large datasets of volumes. To address these challenges, we introduce Raptor (Random Planar Tensor Reduction), a train-free method for generating semantically rich embeddings for volumetric data. Raptor leverages a frozen 2D foundation model, pretrained on natural images, to extract visual tokens from individual cross-sections of medical volumes. These tokens are then spatially compressed using random projections, significantly reducing computational complexity while retaining semantic information. Extensive experiments on ten diverse medical volume tasks verify the superior performance of Raptor over state-of-the-art methods, including those pretrained exclusively on medical volumes (+3% SuPreM, +6% MISFM, +10% Merlin, +13% VoCo, and +14% SLIViT), while entirely bypassing the need for costly training. Our results highlight the effectiveness and versatility of Raptor as a foundation for advancing deep learning-based methods for medical volumes.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Accurate prediction of disease-risk factors from volumetric medical scans by a deep vision model pre-trained with 2D scansNature Biomedical Engineering, 2024
- BrainSegFounder: Towards 3D foundation models for neuroimage segmentationMedical Image Analysis, 2024
- AbdomenAtlas: A large-scale, detailed-annotated, & multi-center dataset for efficient transfer learning and open algorithmic benchmarkingMedical Image Analysis, 2024
- Work like a doctor: Unifying scan localizer and dynamic generator for automated computed tomography report generationExpert Systems with Applications, 2023
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classificationScientific Data, 2023
- The Liver Tumor Segmentation Benchmark (LiTS)Medical Image Analysis, 2022
- Deep-learning-assisted detection and segmentation of rib fractures from CT scans: Development and validation of FracNetEBioMedicine, 2020
- Clinically Applicable AI System for Accurate Diagnosis, Quantitative Measurements, and Prognosis of COVID-19 Pneumonia Using Computed TomographyCell, 2020
- The UK Biobank resource with deep phenotyping and genomic dataNature, 2018
- An elementary proof of a theorem of Johnson and LindenstraussRandom Structures & Algorithms, 2002