Cross-Lingual Text Image Recognition via Multi-Hierarchy Cross-Modal Mimic
- 16 June 2022
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 25 (15209210) , 4830-4841
Abstract
Optical character recognition and machine translation are usually studied and applied separately. In this paper, we consider a new problem named cross-lingual text image recognition (CLTIR) that integrates these two tasks together. The core of this problem is to recognize source language texts shown in images and transcribe them to the target language in an end-to-end manner. Traditional cascaded systems perform text image recognition and text translation sequentially. This can lead to error accumulation and parameter redundancy problems. To overcome these problems, we propose a multihierarchy cross-modal mimic (MHCMM) framework for end-to-end CLTIR, which can be trained with a massive bilingual text corpus and a small number of bilingual annotated text images. In this framework, a plug-in machine translation model is used as a teacher to guide the CLTIR model for learning representations compatible with image and text modes. Via adversarial learning and attention mechanisms, the proposed mimic method can integrate both global and local information in the semantic space. Experiments on a newly collected dataset demonstrate the superiority of the proposed framework. Our method outperforms other pipelines while containing fewer parameters. Additionally, the MHCMM framework can utilize a large-scale bilingual corpus to further improve the performance efficiently. The visualization of attention scores indicates that the proposed model can read text images in a fashion similar to the machine translation model reading text tokens.Keywords
Funding Information
- National Key Research and Development Program of China (2020AAA0108003)
- National Natural Science Foundation of China (61733007, 61721004)
This publication has 36 references indexed in Scilit:
- Simultaneous Script Identification and Handwriting Recognition via Multi-Task Learning of Recurrent Neural Networks2017
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Words Matter: Scene Text for Image Classification and RetrievalIEEE Transactions on Multimedia, 2016
- A Convolutional Neural Network-Based Chinese Text Detection Algorithm via Text Structure ModelingIEEE Transactions on Multimedia, 2016
- Cross Modal Distillation for Supervision Transfer2016
- A New Technique for Multi-Oriented Scene Text Line Detection and Tracking in VideoIEEE Transactions on Multimedia, 2015
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation2014
- A Novel Connectionist System for Unconstrained Handwriting RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2008
- BLEU2001