Track, Attend, and Parse (TAP): An End-to-End Framework for Online Handwritten Mathematical Expression Recognition
- 6 June 2018
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Multimedia
- Vol. 21 (1) , 221-233
Abstract
In this paper, we introduce Track, Attend and Parse (TAP), an end-to-end approach based on neural networks for online handwritten mathematical expression recognition (OHMER). The architecture of TAP consists of a tracker and a parser. The tracker employs a stack of bidirectional recurrent neural networks with gated recurrent units (GRU) to model the input handwritten traces, which can fully utilize the dynamic trajectory information in OHMER. Followed by the tracker, the parser adopts a GRU equipped with guided hybrid attention (GHA) to generate LaTeX notations. The proposed GHA is composed of a coverage based spatial attention, a temporal attention and an attention guider. Moreover, we demonstrate the strong complementarity between offline information with static-image input and online information with ink-trajectory input by blending a fully convolutional networks based watcher into TAP. Inherently unlike traditional methods, this end-to-end framework does not require the explicit symbol segmentation and a predefined expression grammar for parsing. Validated on a benchmark published by the CROHME competition, the proposed approach outperforms the state-of-the-art methods and achieves the best reported results with an expression recognition accuracy of 61.16% on CROHME 2014 and 57.02% on CROHME 2016, using only official training dataset.Keywords
Funding Information
- National Key R&D Program of China (2017YFB1002202)
- National Natural Science Foundation of China (61671422, U1613211)
- Key Science and Technology Project of Anhui Province (17030901005)
- MOE-Microsoft Key Laboratory of USTC
This publication has 44 references indexed in Scilit:
- Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov modelsPattern Recognition Letters, 2014
- A new approach for recognizing handwritten mathematics using relational grammars and fuzzy setsInternational Journal on Document Analysis and Recognition (IJDAR), 2012
- Efficient search strategy in structural analysis for handwritten mathematical expression recognitionPattern Recognition, 2008
- Recognizing mathematical expressions using tree transformationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2002
- Testing the correlation of word error rate and perplexitySpeech Communication, 2002
- Control of goal-directed and stimulus-driven attention in the brainNature Reviews Neuroscience, 2002
- Error detection, error correction and performance evaluation in on-line mathematical expression recognitionPattern Recognition, 2001
- Mathematical expression recognition: a surveyInternational Journal on Document Analysis and Recognition (IJDAR), 2000
- The Dynamic Representation of ScenesVisual Cognition, 2000
- Gauging Similarity with n -Grams: Language-Independent Categorization of TextScience, 1995