TriVLLo: Tri-View Dynamic Architecture and Unified Cross-Modal Representation for Efficient Fine-Grained Vision–Language Understanding

Authors:
Publisher website
Google Scholar
Abstract
This study tackles computational bottlenecks, training instability, and insufficient cross-modal semantic alignment in high-resolution multimodal image processing. We propose TriVLLo, an innovative multi-scale vision-language modeling framework. Our main contributions are: First, we introduce factorized 2D positional encoding and a dynamically configurable modular architecture. This approach decouples height and width position information. It improves spatial localization reliability for images with extreme aspect ratios. It also reduces computational parameters and alleviates training instability. Second, we design a unified multi-scale feature extraction and modality interaction mechanism. This uses adaptive image processing and multi-perspective feature pyramids. It enhances robustness to inputs of any resolution. It also achieves fine-grained alignment of vision-language features through a shared embedding space. Third, we build a high-quality dataset with 11K samples for fine-grained reasoning. This dataset supports improvements in visual ranking, semantic alignment, and narrative reasoning. Experiments show that TriVLLo achieves 87.4% of GPT-4V's performance on the MM-Vet benchmark. It demonstrates a 93.0 percentage-point improvement over Emu2 in spatial cognition tasks. It attains 89.2% accuracy on knowledge generation tasks. These results significantly outperform state-of-the-art methods.
Funding Information
  • Key Technology Research and Development Program of the Zhejiang Province (2022C01125)
  • Zhejiang Province High-Level Talent Special Support Program-Leading Talent of Technological Innovation (2022R52043)

This publication has 11 references indexed in Scilit: