HRU: Harmonizing Reading and Understanding for Parameter-Efficient Scene-Text Aware Cross-Modal Retrieval
Open Access
- 9 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 14626-14635
Abstract
Scene-text aware cross-modal retrieval requires models to jointly capture visual text content and object semantics. However, existing methods often exhibit an inherent imbalance: they either over-rely on fine-grained textual cues and become sensitive to noise, or overlook critical textual information, resulting in suboptimal alignment. To address this issue, we propose HRU, a parameter-efficient framework that Harmonizes Reading and Understanding via a two-stage progressive learning strategy and orthogonality constraints. In the first stage, visual prompts disentangle object-level and text-level representations, enabling adaptive scene text encoding while preserving visual semantics. In the second stage, a fusion mechanism progressively aligns and refines these features, achieving coherent heterogeneous visual information integration. With a parameter-efficient design, HRU can be seamlessly adapted to multimodal backbones of varying scales with minimal fine-tuning overhead. Extensive experiments demonstrate that HRU significantly improves alignment and retrieval robustness, achieving RSUM scores of 474.5, 380.3, and 546.3 on CTC-1k, CTC-5k, and TextCaps, respectively, with fewer than 30K additional trainable parameters. These results establish HRU as an effective and unified solution for text-rich visual understanding and retrieval.Keywords
Funding Information
- Basic research program of Jiangsu (BK20252040)
- Basic research program of Jiangsu (BK20251441)
- Basic research program of Jiangsu (BK20251414)
- National Natural Science Foundation of China (62302223)
- Guangdong Basic and Applied Basic Research Foundation (2025A1515011224)
This publication has 14 references indexed in Scilit:
- Heterogeneous Prompt-Guided Entity Inferring and Distilling for Scene-Text Aware Cross-Modal RetrievalProceedings of the AAAI Conference on Artificial Intelligence, 2025
- Fine-grained extraction of geospatial and temporal information from Chinese historical newspapersDigital Scholarship in the Humanities, 2025
- Focus, Distinguish, and Prompt: Unleashing CLIP for Efficient and Flexible Scene Text Retrieval2024
- Scene-text aware cross-modal retrieval based on semantic matching (ChinaMM2024)Multimedia Systems, 2024
- Granularity-Aware Single-Point Scene Text Spotting With Sequential Recurrence Self-AttentionIEEE Transactions on Circuits and Systems for Video Technology, 2024
- Turning a CLIP Model Into a Scene Text SpotterIEEE Transactions on Pattern Analysis and Machine Intelligence, 2024
- Visual Prompt Tuning2022
- OCR-Free Document Understanding Transformer2022
- Integrating Scene Text and Visual Appearance for Fine-Grained Image ClassificationIEEE Access, 2018
- Con-Text: Text Detection for Fine-Grained Object ClassificationIEEE Transactions on Image Processing, 2017