LOCR: Location-Guided Transformer for Optical Character Recognition
Authors:
Preprint
Abstract
Academic documents are packed with texts, equations, tables, and figures, requiring comprehensive understanding for accurate Optical Character Recognition (OCR). While end-to-end OCR methods offer improved accuracy over layout-based approaches, they often grapple with significant repetition issues, especially with complex layouts in Out-Of-Domain (OOD) documents.To tackle this issue, we propose LOCR, a model that integrates location guiding into the transformer architecture during autoregression. We train the model on a dataset comprising over 77M text-location pairs from 125K academic document pages, including bounding boxes for words, tables and mathematical symbols. LOCR adeptly handles various formatting elements and generates content in Markdown language. It outperforms all existing methods in our test set constructed from arXiv, as measured by edit distance, BLEU, METEOR and F-measure.LOCR also reduces repetition frequency from 4.4% of pages to 0.5% in the arXiv dataset, from 13.2% to 1.3% in OOD quantum physics documents and from 8.1% to 1.8% in OOD marketing documents. Additionally, LOCR features an interactive OCR mode, facilitating the generation of complex documents through a few location prompts from human.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- Vary: Scaling up the Vision Vocabulary for Large Vision-Language Model2024
- Visual in-Context Prompting2024
- Recent trends in mathematical expressions recognition: An LDA-based analysisExpert Systems with Applications, 2022
- LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking2022
- DocBed: A Multi-Stage OCR Solution for Documents with Complex LayoutsProceedings of the AAAI Conference on Artificial Intelligence, 2022
- LayoutLM: Pre-training of Text and Layout for Document Image Understanding2020
- Full-Page Text Recognition: Learning Where to Start and When to Stop2017
- EAST: An Efficient and Accurate Scene Text Detector2017
- An Overview of the Tesseract OCR EngineNinth International Conference on Document Analysis and Recognition (ICDAR 2007), 2007
- Best practices for convolutional neural networks applied to visual document analysis2005