mmLayout: Multi-grained MultiModal Transformer for Document Understanding
Authors:
Open Access
- 10 October 2022
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 4877-4886
Abstract
Recent efforts of multimodal Transformers have improved Visually Rich Document Understanding (VrDU) tasks via incorporating visual and textual information. However, existing approaches mainly focus on fine-grained elements such as words and document image patches, making it hard for them to learn from coarse-grained elements, including natural lexical units like phrases and salient visual regions like prominent image regions. In this paper, we attach more importance to coarse-grained elements containing high-density information and consistent semantics, which are valuable for document understanding. At first, a document graph is proposed to model complex relationships among multi-grained multimodal elements, in which salient visual regions are detected by a cluster-based method. Then, a multi-grained multimodal Transformer called mmLayout is proposed to incorporate coarse-grained information into existing pre-trained fine-grained multimodal Transformers based on the graph. In mmLayout, coarse-grained information is aggregated from fine-grained, and then, after further processing, is fused back into fine-grained for final prediction. Furthermore, common sense enhancement is introduced to exploit the semantic information of natural lexical units. Experimental results on four tasks, including information extraction and document question answering, show that our method can improve the performance of multimodal Transformers based on fine-grained elements and achieve better performance with fewer parameters. Qualitative analyses show that our method can capture consistent semantics in coarse-grained elements.Keywords
All Related Versions
Funding Information
- NSFC (No. 62072399 No. U19B2042 No. 61402403)
- Chinese Knowledge Center for Engineering Sciences and Technology
- Fundamental Research Funds for the Central Universities (No. 226-2022-00070)
- National Key R&D Program of China (No. 2018AAA0101900)
- MoE Engineering Research Center of Digital Library
This publication has 13 references indexed in Scilit:
- ViBERTgrid: A Jointly Trained Multi-modal 2D Document Representation for Key Information Extraction from Documents2021
- LayoutLMv2: Multi-modal Pre-training for Visually-rich Document Understanding2021
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding2020
- DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding2020
- Representation Learning for Information Extraction from Form-like Documents2020
- Deterministic Routing between Layout Abstractions for Multi-Scale Classification of Visually Rich Documents2019
- Graph Convolution for Multimodal Information Extraction from Visually Rich Documents2019
- Stacked Cross Attention for Image-Text Matching2018
- Chargrid: Towards Understanding 2D Documents2018
- Feature Pyramid Networks for Object Detection2017