Text-guided Vision Token Reduction with Low-rank Adaptation for Efficient Visual Grounding
- 16 April 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Circuits and Systems for Video Technology
- Vol. PP (10518215) , 1
Abstract
Transformer-based pretrained models have advanced Visual Grounding (VG) significantly, but their scaling up has caused soaring training and inference costs. While efforts adapting Parameter-Efficient Fine-Tuning to VG have cut training expenses to some extent, inference costs are over-looked, stemming from Transformer’s computation costs growing quadratically with input token length. To address the above challenge, we propose a text-guided token prune method based on a one-stream architecture for VG, named OneSVG. Specifically, OneSVG transfers low-rank adaptation LoRA to VG, and calculates the relevance between each vision token and text semantics in multiple stages, and gradually prunes the vision tokens with low relevance. In addition, traditional VG methods use a single [REG] token to predict bounding boxes, and the [REG] token relies on complete vision tokens during training, pruning large number of vision tokens will disrupt the spatial orientation information of images. Therefore, the prediction head in OneSVG first restores the square structure of images by padding the missing area, and then accepts complete vision tokens. Experimental results on three widely-used benchmarks demonstrate that our OneSVG achieves state-of-the-art real-time speed while maintaining the best accuracy with only 34.3% tokens. Codes and models are available at https://github.com/ltShi/OneSVG.Keywords
Funding Information
- National Natural Science Foundation of China (U23B2031)
This publication has 58 references indexed in Scilit:
- YORO - Lightweight End to End Visual Grounding2023
- SeqTR: A Simple Yet Universal Network for Visual Grounding2022
- Visual Prompt Tuning2022
- Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework2022
- Improving One-Stage Visual Grounding by Recursive Sub-query Construction2020
- Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression2019
- Embodied Question Answering2018
- Modeling Context in Referring Expressions2016
- Modeling Context Between Objects for Referring Expression Understanding2016
- Generation and Comprehension of Unambiguous Object Descriptions2016