Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Preprint
Abstract
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.Keywords
All Related Versions
This publication has 24 references indexed in Scilit:
- TextMonkey: An OCR-Free Large Multimodal Model for Understanding DocumentIEEE Transactions on Pattern Analysis and Machine Intelligence, 2026
- MulCogBench: a multi-modal cognitive benchmark dataset for evaluating Chinese and English computational language modelsLanguage Resources and Evaluation, 2025
- A simple clustering approach to map the human brain's cortical semantic network organization during taskNeuroImage, 2025
- Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image TranslationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2025
- Vary: Scaling up the Vision Vocabulary for Large Vision-Language Model2024
- Multi-teacher Knowledge Distillation for End-to-End Text Image Machine Translation2023
- Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task2022
- InfographicVQA2022
- DocVQA: A Dataset for VQA on Document Images2021
- ICDAR 2015 competition on Robust Reading2015