Representing Part-Whole Hierarchies in Foundation Models by Learning Localizability, Composability, and Decomposability from Anatomy via Self-Supervision
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE) in Conference on Computer Vision and Pattern Recognition (CVPR)
- Vol. abs/210504 (2024) , 11269-11281
Abstract
Humans effortlessly interpret images by parsing them into part-whole hierarchies; deep learning excels in learning multi-level feature spaces, but they often lack explicit coding of part-whole relations, a prominent property of medical imaging. To overcome this limitation, we introduce Adam-v2, a new self-supervised learning framework extending Adam [69] by explicitly incorporating part-whole hierarchies into its learning objectives through three key branches: (1) Localizability, acquiring discriminative representations to distinguish different anatomical patterns; (2) Composability, learning each anatomical structure in a parts-to-whole manner; and (3) Decomposability, comprehending each anatomical structure in a whole-to-parts manner. Experimental results across 10 tasks, compared to 11 baselines in zero-shot, few-shot transfer, and full fine-tuning settings, showcase Adam-v2's superior performance over large-scale medical models and existing SSL methods across diverse downstream tasks. The higher generality and robustness of Adam-v2's representations originate from its explicit construction of hierarchies for distinct anatomical structures from unlabeled medical images. Adam-v2 preserves a semantic balance of anatomical diversity and harmony in its embedding, yielding representations that are both generic and semantically meaningful, yet overlooked in existing SSL methods. All code and pretrained models are available at GitHub.com/JLiangLab/Eden.Keywords
All Related Versions
This publication has 72 references indexed in Scilit:
- Towards Foundation Models Learned from Anatomy in Medical Imaging via Self-supervision2023
- Anatomy-Aware Contrastive Representation Learning for Fetal Ultrasound2023
- RegionCL: Exploring Contrastive Region Pairs for Self-supervised Representation Learning2022
- Discriminative, Restorative, and Adversarial Learning: Stepwise Incremental Pretraining2022
- POPAR: Patch Order Prediction and Appearance Recovery for Self-supervised Medical Image Analysis2022
- Learning Semantics-Enriched Representation via Self-discovery, Self-classification, and Self-restoration2020
- Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis2019
- U-Net: Convolutional Networks for Biomedical Image Segmentation2015
- Segmentation of anatomical structures in chest radiographs using supervised methods: a comparative study on a public databaseMedical Image Analysis, 2006
- Mapping part-whole hierarchies into connectionist networksArtificial Intelligence, 1990