AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 12490-12500
Abstract
A handful of visual foundation models (VFMs) have recently emerged as the backbones for numerous downstream tasks. VFMs like CLIP, DINOv2, SAM are trained with distinct objectives, exhibiting unique characteristics for various downstream tasks. We find that despite their conceptual differences, these models can be effectively merged into a unified model through multi-teacher distillation. We name this approach AM-RADIO (Agglomerative Model - Reduce All Domains Into One). This integrative approach not only surpasses the performance of individual teacher models but also amalgamates their distinctive features, such as zero-shot vision-language comprehension, detailed pixel-level understanding, and open vocabulary segmentation capabilities. Additionally, in pursuit of the most hardware-efficient backbone, we evaluated numerous architectures in our multi-teacher distillation pipeline using the same training recipe. This led to the development of a novel architecture (E-RADIO) that exceeds the performance of its predecessors and is at least 6x faster than the teacher models at matched resolution. Our comprehensive benchmarking process covers downstream tasks including ImageNet classification, semantic segmentation linear probing, COCO object detection and integration into LLaVa-1.5. Code: https://github.com/NVlabs/RADIO.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- MaxViT: Multi-axis Vision Transformer2022
- EdgeViTs: Competing Light-Weight CNNs on Mobile Devices with Vision Transformers2022
- Adaptive multi-teacher multi-level knowledge distillationNeurocomputing, 2020
- Towards VQA Models That Can Read2019
- Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics2018
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering2017
- Scene Parsing through ADE20K Dataset2017
- The Pascal Visual Object Classes Challenge: A RetrospectiveInternational Journal of Computer Vision, 2014
- ImageNet: A large-scale hierarchical image database2009