Multi-Teacher Knowledge Distillation For Text Image Machine Translation
Preprint
Abstract
Text image machine translation (TIMT) has been widely used in various real-world applications, which translates source language texts in images into another target language sentence. Existing methods on TIMT are mainly divided into two categories: the recognition-then-translation pipeline model and the end-to-end model. However, how to transfer knowledge from the pipeline model into the end-to-end model remains an unsolved problem. In this paper, we propose a novel Multi-Teacher Knowledge Distillation (MTKD) method to effectively distillate knowledge into the end-to-end TIMT model from the pipeline model. Specifically, three teachers are utilized to improve the performance of the end-to-end TIMT model. The image encoder in the end-to-end TIMT model is optimized with the knowledge distillation guidance from the recognition teacher encoder, while the sequential encoder and decoder are improved by transferring knowledge from the translation sequential and decoder teacher models. Furthermore, both token and sentence-level knowledge distillations are incorporated to better boost the translation performance. Extensive experimental results show that our proposed MTKD effectively improves the text image translation performance and outperforms existing end-to-end and pipeline models with fewer parameters and less decoding time, illustrating that MTKD can take advantage of both pipeline and end-to-end models.Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- Cross-Lingual Text Image Recognition via Multi-Hierarchy Cross-Modal MimicIEEE Transactions on Multimedia, 2022
- Optical Character Recognition and Neural Machine Translation Using Deep Learning Techniques2021
- Robust Text Image Recognition via Adversarial Sequence-to-Sequence Domain AdaptationIEEE Transactions on Image Processing, 2021
- An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text RecognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Robust Scene Text Recognition with Automatic Rectification2016
- Integrating natural language processing with image document analysis: what we learned from two real-world applicationsInternational Journal on Document Analysis and Recognition (IJDAR), 2015
- Cyclops – Snapshot Translation System Based on Mobile DeviceJournal of Software, 2011
- An image-based automatic Arabic translation systemPattern Recognition, 2009