AdaMV-MoE: Adaptive Multi-Task Vision Mixture-of-Experts
Authors:
- 1 October 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 17300-17311
Abstract
Sparsely activated Mixture-of-Experts (MoE) is becoming a promising paradigm for multi-task learning (MTL). Instead of compressing multiple tasks’ knowledge into a single model, MoE separates the parameter space and only utilizes the relevant model pieces given task type and its input, which provides stabilized MTL training and ultra-efficient inference. However, current MoE approaches adopt a fixed network capacity (e.g., two experts in usual) for all tasks. It potentially results in the over-fitting of simple tasks or the under-fitting of challenging scenarios, especially when tasks are significantly distinctive in their complexity. In this paper, we propose an adaptive MoE framework for multi-task vision recognition, dubbed AdaMV-MoE. Based on the training dynamics, it automatically determines the number of activated experts for each task, avoiding the laborious manual tuning of optimal model size. To validate our proposal, we benchmark it on ImageNet classification and COCO object detection & instance segmentation which are notoriously difficult to learn in concert, due to their discrepancy. Extensive experiments across a variety of vision transformers demonstrate a superior performance of AdaMV-MoE, compared to MTL with a shared backbone and the recent state-of-the-art (SoTA) MTL MoE approach. Codes are available online: https://github.com/google-research/google-research/tree/master/moe_mtl.Keywords
This publication has 52 references indexed in Scilit:
- A Simple Single-Scale Vision Transformer for Object Detection and Instance Segmentation2022
- End-to-End Object Detection with Transformers2020
- MTI-Net: Multi-scale Task Interaction Networks for Multi-task Learning2020
- Revisiting RCNN: On Awakening the Classification Power of Faster RCNN2018
- Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation2018
- Deep Networks with Stochastic Depth2016
- Network of Experts for Large-Scale Image Categorization2016
- MODULAR ARCHITECTURE2011
- ImageNet: A large-scale hierarchical image database2009
- Improved learning algorithms for mixture of experts in multiclass classificationNeural Networks, 1999