Bridging the Gap with Convolutional networks: A Graph-based Vision Transformer with Sparsity for Training on Small Datasets from Scratch
Authors:
Preprint
- 14 November 2024
- preprint
- Published by Springer Nature in Research Square
Abstract
Vision Transformers (ViTs) have achieved impressive results in large-scale image classification. However, when training from scratch on small datasets, there is still a significant performance gap between ViTs and Convolutional Neural Networks (CNNs), which is attributed to the lack of inductive bias. To address this issue, we propose a Graph-based Vision Transformer (GvT) that utilizes graph convolutional projection and graphpooling. In each block, queries and keys are calculated through graph convolutional projection based on the spatial adjacency matrix, while dot-product attention is used in another graph convolution to generate values. When using more attention heads, the queries and keys become lower-dimensional, making their dot product an uninformative matching function. To overcome this low-rank bottleneck in attention heads, we employ talkingheads technology based on bilinear pooled features and sparse selection of attention tensors. This allows interaction among filtered attention scores and enables each attention mechanism to depend on all queries and keys. Additionally, we apply graphpooling between two intermediate blocks to reduce the number of tokens and aggregate semantic information more effectively. Our experimental results show that GvT produces comparable or superior outcomes to deep convolutional networks and surpasses vision transformers without pre-training on large datasets. The code for our proposed model is publicly available on the website: GitHub/GvT.Keywords
All Related Versions
This publication has 4 references indexed in Scilit:
- CvT: Introducing Convolutions to Vision Transformers2021
- DRRNets: Dynamic Recurrent Routing via Low-Rank Regularization in Recurrent Neural NetworksIEEE Transactions on Neural Networks and Learning Systems, 2021
- Second-Order Pooling for Graph Neural NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
- Identifying pneumonia in chest X-rays: A deep learning approachMeasurement, 2019