Yu Zhou
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Yu Zhou has a total of 51 co-authors. Here are the top co-authors by number of shared publications:
- Chengqing Zong
Institute of Automation, Chinese Academy of Sciences
26 shared publications - Yang Zhao
Institute of Automation, Chinese Academy of Sciences
17 shared publications - Yaping Zhang
State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation CAS Beijing China
16 shared publications Lu Xiang
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences, Beijing, China
10 shared publications- Zhiyang Zhang
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China
10 shared publications Yupu Liang
Institute of Automation State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS) Chinese Academy of Sciences Beijing China
9 shared publications
Latest Publications
- 13 January 2026
- Published by Springer Nature
- p. 125-139
Boosting Document Image Translation via Layout-Aware Semantic Paragraph Clustering
- 13 January 2026
- Published by Springer Nature
- p. 125-139
Boosting Document Image Translation via Layout-Aware Semantic Paragraph Clustering
- 4 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4736-4747
Imitate, Reward and Introspect: Enhancing Neural Machine Translation by Utilizing Large Language Models as Environments
The current neural machine translation (NMT) models face several challenges including the high cost of parallel sentence pairs and difficulties in learning from limited error examples. To address these issues, we propose a novel framework, Large Language Models as Environments (LLMaE), which leverages interactions with large language models (LLMs) to enhance NMT performance. The LLMaE framework comprises three logically sequential steps: behavior imitation, reward feedback, and introspection. The behavior imitation step facilitates the NMT model to mimic the behavioral patterns of the LLMs. The reward feedback step leverages the quality assessments provided by the LLMs to optimize the NMT model and encourage it to produce more accurate translations. Finally, the introspection step enables the NMT model to learn from limited error examples by generating introspective examples. Each stage builds on the previous one, creating a cycle of continuous improvement. Extensive experimental results demonstrate that our proposed method significantly improves the performance of NMT in various scenarios and with different LLMs.
- 4 November 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 4736-4747
Imitate, Reward and Introspect: Enhancing Neural Machine Translation by Utilizing Large Language Models as Environments
The current neural machine translation (NMT) models face several challenges including the high cost of parallel sentence pairs and difficulties in learning from limited error examples. To address these issues, we propose a novel framework, Large Language Models as Environments (LLMaE), which leverages interactions with large language models (LLMs) to enhance NMT performance. The LLMaE framework comprises three logically sequential steps: behavior imitation, reward feedback, and introspection. The behavior imitation step facilitates the NMT model to mimic the behavioral patterns of the LLMs. The reward feedback step leverages the quality assessments provided by the LLMs to optimize the NMT model and encourage it to produce more accurate translations. Finally, the introspection step enables the NMT model to learn from limited error examples by generating introspective examples. Each stage builds on the previous one, creating a cycle of continuous improvement. Extensive experimental results demonstrate that our proposed method significantly improves the performance of NMT in various scenarios and with different LLMs.
- 16 September 2025
- Published by Springer Nature
- p. 505-522
ICDAR 2025 Competition on End-to-End Document Image Machine Translation Towards Complex Layouts
- 16 September 2025
- Published by Springer Nature
- p. 505-522
ICDAR 2025 Competition on End-to-End Document Image Machine Translation Towards Complex Layouts
- Open Access
Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.
Open AccessImproving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks.
- Open Access
Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios.
Open AccessSingle-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios.
- 11 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2606-2621
Reading When Translating: Multi-Modal Document Image Machine Translation With Reading Flow Prediction
Document Image Translation (DIT) aims to translate documents in images from one language to another. It is a multi-modal task that involves the cooperation of text, visual layout, and reading logical order. However, existing text-based or vision-based methods rely on solely textual or visual features. Layout-based methods are multi-modal but largely overlook the crucial reading logical order. To fully leverage the multi-modal information and exploit explicit modules to learn better reading logical order for DIT, this paper proposes the “reading-when-translating” guideline. It collaborates the translation process with an auxiliary “reading” process such that reading logical order directly contributes to translation. Following this guideline, we propose Document Reading and Translation Network (DocRTN), a novel unified framework that seamlessly integrates reading order into DIT, enabling the model to handle complex layouts by “reading” text in a human-like, coherent sequence. The unified framework comprises a reading flow decoder and a translation decoder. A novel feature decorator is proposed to harmonize the reading and translation channels, ensuring reading features are optimally adapted for translation. Extensive comparisons and analysis on 5 domains and 4 translation directions demonstrate DocRTN outperforms previous state-of-the-arts in all aspects.
- 11 June 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Audio, Speech and Language Processing
- Vol. 33, 2606-2621
Reading When Translating: Multi-Modal Document Image Machine Translation With Reading Flow Prediction
Document Image Translation (DIT) aims to translate documents in images from one language to another. It is a multi-modal task that involves the cooperation of text, visual layout, and reading logical order. However, existing text-based or vision-based methods rely on solely textual or visual features. Layout-based methods are multi-modal but largely overlook the crucial reading logical order. To fully leverage the multi-modal information and exploit explicit modules to learn better reading logical order for DIT, this paper proposes the “reading-when-translating” guideline. It collaborates the translation process with an auxiliary “reading” process such that reading logical order directly contributes to translation. Following this guideline, we propose Document Reading and Translation Network (DocRTN), a novel unified framework that seamlessly integrates reading order into DIT, enabling the model to handle complex layouts by “reading” text in a human-like, coherent sequence. The unified framework comprises a reading flow decoder and a translation decoder. A novel feature decorator is proposed to harmonize the reading and translation channels, ensuring reading features are optimally adapted for translation. Extensive comparisons and analysis on 5 domains and 4 translation directions demonstrate DocRTN outperforms previous state-of-the-arts in all aspects.
- 17 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (5) , 3358-3376
Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation
Document Image Translation (DIT) aims to translate texts on document images from one language to another. It is a multi-modal task involving cooperation of text and layout. Current approaches either handle layout and translation as separate processes, risking accumulative errors, or use vanilla end-to-end encoder-decoder models to capture layout implicitly, often suffering inadequate layout incorporation. We argue that a favorable framework should explicitly engage layout-specific modules and properly organize them toward translation. For this, we first revisit two key layouts: the geometric layout reflecting word's spatial positions, and the logical layout depicting word's logical order. Then, a novel pipeline (understand layout $\rightarrow$ translate text) is determined to prioritize layouts such that preceding layouts contribute to translation. Following this pipeline, we introduce Unified Document Image Translation (UniDIT), a comprehensive framework that unifies layout with translation in one network. It is devised to leverage each module's advantage, and provide an elaborate feature-conductive flow for module communication globally. A novel bridging mechanism is also introduced to adapt layout features conducive to translation. We further contribute DITransv2, a large-scale fine-grained benchmark that includes heterogeneous and complex document layouts. Extensive experiments on DITransv2 and additional established benchmarks demonstrate UniDIT outperforms previous state-of-the-arts in all aspects.
- 17 January 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. 47 (5) , 3358-3376
Understand Layout and Translate Text: Unified Feature-Conductive End-to-End Document Image Translation
Document Image Translation (DIT) aims to translate texts on document images from one language to another. It is a multi-modal task involving cooperation of text and layout. Current approaches either handle layout and translation as separate processes, risking accumulative errors, or use vanilla end-to-end encoder-decoder models to capture layout implicitly, often suffering inadequate layout incorporation. We argue that a favorable framework should explicitly engage layout-specific modules and properly organize them toward translation. For this, we first revisit two key layouts: the geometric layout reflecting word's spatial positions, and the logical layout depicting word's logical order. Then, a novel pipeline (understand layout $\rightarrow$ translate text) is determined to prioritize layouts such that preceding layouts contribute to translation. Following this pipeline, we introduce Unified Document Image Translation (UniDIT), a comprehensive framework that unifies layout with translation in one network. It is devised to leverage each module's advantage, and provide an elaborate feature-conductive flow for module communication globally. A novel bridging mechanism is also introduced to adapt layout features conducive to translation. We further contribute DITransv2, a large-scale fine-grained benchmark that includes heterogeneous and complex document layouts. Extensive experiments on DITransv2 and additional established benchmarks demonstrate UniDIT outperforms previous state-of-the-arts in all aspects.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 9767-9783
CROP: Contextual Region-Oriented Visual Token Pruning
Jiawei Guo, Feifei Zhai, Pu Jian, Qianrun Wei, Yu Zhou. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 9767-9783
Open AccessCROP: Contextual Region-Oriented Visual Token Pruning
Jiawei Guo, Feifei Zhai, Pu Jian, Qianrun Wei, Yu Zhou. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 23659-23678
Improving MLLM’s Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Yupu Liang, Yaping Zhang, Zhiyang Zhang, Zhiyuan Chen, Yang Zhao, Lu Xiang, Chengqing Zong, Yu Zhou. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 23659-23678
Open AccessImproving MLLM’s Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Yupu Liang, Yaping Zhang, Zhiyang Zhang, Zhiyuan Chen, Yang Zhao, Lu Xiang, Chengqing Zong, Yu Zhou. Findings of the Association for Computational Linguistics: ACL 2025. 2025.
- Open Access
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 11755-11771
TROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.
- 1 January 2025
- Published by Association for Computational Linguistics (ACL)
- p. 11755-11771
Open AccessTROVE: A Challenge for Fine-Grained Text Provenance via Source Sentence Tracing and Relationship Classification
Junnan Zhu, Min Xiao, Yining Wang, Feifei Zhai, Yu Zhou, Chengqing Zong. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.