OmniParser V2: Structured-Points-of-Thought for Unified Visual Text Parsing and Its Generality to Multimodal Large Language Models
Top Cited Papers
- 24 March 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 48 (8) , 9210-9227
Abstract
Visually-situated text parsing (VsTP) has recently seen notable advancements, driven by the growing demand for automated document understanding and the emergence of large language models capable of processing document-based questions. While various methods have been proposed to tackle the complexities of VsTP, existing solutions often rely on task-specific architectures and objectives for individual tasks. This leads to modal isolation and complex workflows due to the diversified targets and heterogeneous schemas. In this paper, we introduce OmniParser V2, a universal model that unifies VsTP typical tasks, including text spotting, key information extraction, table recognition, and layout analysis, into a unified framework. Central to our approach is the proposed Structured-Points-of-Thought (SPOT) prompting schemas, which improves model performance across diverse scenarios by leveraging a unified encoder-decoder architecture, objective, and input&output representation. SPOT eliminates the need for task-specific architectures and loss functions, significantly simplifying the processing pipeline. Our extensive evaluations across four tasks on eight different datasets show that OmniParser V2 achieves state-of-the-art or competitive results in VsTP. Additionally, we explore the integration of SPOT within a multimodal large language model structure, further enhancing visual text parsing capabilities on four tasks, thereby confirming the generality of SPOT prompting technique. The code is available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery.Keywords
All Related Versions
Funding Information
- National Natural Science Foundation of China (62225603)
- National Natural Science Foundation of China (62206104)
- National Key Research and Development Program of China (2022YFC2305102)
- Alibaba Innovative Research
This publication has 122 references indexed in Scilit:
- GLASS: Global to Local Attention for Scene-Text Spotting2022
- Language Matters: A Weakly Supervised Vision-Language Pre-training Approach for Scene Text Detection and Spotting2022
- OCR-Free Document Understanding Transformer2022
- ICDAR 2021 Competition on Scientific Table Image Recognition to LaTeX2021
- Mask TextSpotter v3: Segmentation Proposal Network for Robust Scene Text Spotting2020
- Image-Based Table Recognition: Data, Model, and Evaluation2020
- End-to-End Object Detection with Transformers2020
- Table Structure Recognition Using Top-Down and Bottom-Up Cues2020
- Character Region Attention for Text Spotting2020
- Mask TextSpotter: An End-to-End Trainable Neural Network for Spotting Text with Arbitrary Shapes2018