Automatic Metric Validation for Grammatical Error Correction
Preprint
Abstract
Metric validation in Grammatical Error Correction (GEC) is currently done by observing the correlation between human and metric-induced rankings. However, such correlation studies are costly, methodologically troublesome, and suffer from low inter-rater agreement. We propose MAEGE, an automatic methodology for GEC metric validation, that overcomes many of the difficulties with existing practices. Experiments with \maege\ shed a new light on metric quality, showing for example that the standard $M^2$ metric fares poorly on corpus-level ranking. Moreover, we use MAEGE to perform a detailed analysis of metric behavior, showing that correcting some types of errors is consistently penalized by existing metrics.
Keywords
All Related Versions
This publication has 8 references indexed in Scilit:
- SeqGAN: Sequence Generative Adversarial Nets with Policy GradientProceedings of the AAAI Conference on Artificial Intelligence, 2017
- Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image AnnotationsInternational Journal of Computer Vision, 2017
- Reassessing the Goals of Grammatical Error Correction: Fluency Instead of GrammaticalityTransactions of the Association for Computational Linguistics, 2016
- Optimizing Statistical Machine Translation for Text SimplificationTransactions of the Association for Computational Linguistics, 2016
- Evaluating Human Pairwise Preference JudgmentsComputational Linguistics, 2015
- Human Evaluation of Grammatical Error Correction Systems2015
- How Far are We from Fully Automatic High Quality Grammatical Error Correction?2015
- Towards a standard evaluation method for grammatical error detection and correction2015