Parameter Efficient Fine-Tuning via Cross Block Orchestration for Segment Anything Model
Authors:
- 16 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 3743-3752
Abstract
Parameter-efficient fine-tuning (PEFT) is an effective methodology to unleash the potential of large foundation models in novel scenarios with limited training data. In the computer vision community, PEFT has shown effectiveness in image classification, but little research has studied its ability for image segmentation. Fine-tuning segmentation models usually requires a heavier adjustment of parameters to align the proper projection directions in the parameter space for new scenarios. This raises a challenge to existing PEFT algorithms, as they often inject a limited number of individual parameters into each block, which prevents substantial adjustment of the projection direction of the parameter space due to the limitation of Hidden Markov Chain along blocks. In this paper, we equip PEFT with a cross-block orchestration mechanism to enable the adaptation of the Segment Anything Model (SAM) to various downstream scenarios. We introduce a novel inter-block communication module, which integrates a learnable relation matrix to facilitate communication among different coefficient sets of each PEFT block's parameter space. Moreover, we propose an intra-block enhancement module, which introduces a linear projection head whose weights are generated from a hyper-complex layer, further enhancing the impact of the adjustment of projection directions on the entire parameter space. Extensive experiments on diverse benchmarks demonstrate that our proposed approach consistently improves the segmentation performance significantly on novel scenarios with only around 1K additional parameters.Keywords
All Related Versions
Funding Information
- NSFC (62322604,62176159)
- Natural Science Foundation of Shanghai Municipality (21ZR1432200)
This publication has 29 references indexed in Scilit:
- Visual Prompt Tuning2022
- The Medical Segmentation DecathlonNature Communications, 2022
- Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers2021
- Dataset of breast ultrasound imagesData in Brief, 2020
- On the information bottleneck theory of deep learning*Journal of Statistical Mechanics: Theory and Experiment, 2019
- A survey on object detection in optical remote sensing imagesISPRS Journal of Photogrammetry and Remote Sensing, 2016
- Tensor–tensor products with invertible linear transformsLinear Algebra and its Applications, 2015
- Multi-class geospatial object detection and geographic image classification based on collection of part detectorsISPRS Journal of Photogrammetry and Remote Sensing, 2014
- Microsoft COCO: Common Objects in Context2014
- Hypercomplex numbers, lie groups, and the creation of group representation theoryArchive for History of Exact Sciences, 1972