EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm
Preprint
Abstract
Motivated by biological evolution, this paper explains the rationality of Vision Transformer by analogy with the proven practical Evolutionary Algorithm (EA) and derives that both have consistent mathematical formulation. Then inspired by effective EA variants, we propose a novel pyramid EATFormer backbone that only contains the proposed \emph{EA-based Transformer} (EAT) block, which consists of three residual parts, i.e., \emph{Multi-Scale Region Aggregation} (MSRA), \emph{Global and Local Interaction} (GLI), and \emph{Feed-Forward Network} (FFN) modules, to model multi-scale, interactive, and individual information separately. Moreover, we design a \emph{Task-Related Head} (TRH) docked with transformer backbone to complete final information fusion more flexibly and \emph{improve} a \emph{Modulated Deformable MSA} (MD-MSA) to dynamically model irregular locations. Massive quantitative and quantitative experiments on image classification, downstream tasks, and explanatory experiments demonstrate the effectiveness and superiority of our approach over State-Of-The-Art (SOTA) methods. \Eg, our Mobile (1.8M), Tiny (6.1M), Small (24.3M), and Base (49.0M) models achieve 69.4, 78.4, 83.1, and 83.9 Top-1 only trained on ImageNet-1K with naive training recipe; EATFormer-Tiny/Small/Base armed Mask-R-CNN obtain 45.4/47.4/49.0 box AP and 41.4/42.9/44.2 mask AP on COCO detection, surpassing contemporary MPViT-T, Swin-T, and Swin-S by 0.6/1.4/0.5 box AP and 0.4/1.3/0.9 mask AP separately with less FLOPs; Our EATFormer-Small/Base achieve 47.3/49.3 mIoU on ADE20K by Upernet that exceeds Swin-T/S by 2.8/1.7. Code is available at \url{https://github.com/zhangzjn/EATFormer}.Keywords
All Related Versions
This publication has 87 references indexed in Scilit:
- EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications2023
- MaxViT: Multi-axis Vision Transformer2022
- Medical Transformer: Gated Axial-Attention for Medical Image Segmentation2021
- Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based LocalizationInternational Journal of Computer Vision, 2019
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Unified Perceptual Parsing for Scene Understanding2018
- Microsoft COCO: Common Objects in Context2014
- Multi-Population Evolutionary Algorithm for Solving Constrained Optimization Problems2006
- A Fuzzy Adaptive Differential Evolution AlgorithmSoft Computing, 2004
- Performance Scaling of Multi-objective Evolutionary Algorithms2003