ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy
Authors:
Preprint
Abstract
Modern computer vision offers a great variety of models to practitioners, and selecting a model from multiple options for specific applications can be challenging. Conventionally, competing model architectures and training protocols are compared by their classification accuracy on ImageNet. However, this single metric does not fully capture performance nuances critical for specialized tasks. In this work, we conduct an in-depth comparative analysis of model behaviors beyond ImageNet accuracy, for both ConvNet and Vision Transformer architectures, each across supervised and CLIP training paradigms. Although our selected models have similar ImageNet accuracies and compute requirements, we find that they differ in many other aspects: types of mistakes, output calibration, transferability, and feature invariance, among others. This diversity in model characteristics, not captured by traditional metrics, highlights the need for more nuanced analysis when choosing among different models. Our code is available at https://github.com/kirill-vish/Beyond-INet.Keywords
All Related Versions
This publication has 20 references indexed in Scilit:
- DeiT III: Revenge of the ViT2022
- An Impartial Take to the CNN vs Transformer Robustness Contest2022
- CLIP4Clip: An empirical study of CLIP for end to end video clip retrieval and captioningNeurocomputing, 2022
- Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels2021
- On Robustness and Transferability of Convolutional Neural Networks2021
- Natural Adversarial Examples2021
- Big Transfer (BiT): General Visual Representation Learning2020
- Momentum Contrast for Unsupervised Visual Representation Learning2020
- Do Better ImageNet Models Transfer Better?2019
- Deep Residual Learning for Image Recognition2016