CLEME: Debiasing Multi-reference Evaluation for Grammatical Error Correction
Preprint
Abstract
It is intractable to evaluate the performance of Grammatical Error Correction (GEC) systems since GEC is a highly subjective task. Designing an evaluation metric that is as objective as possible is crucial to the development of GEC task. Previous mainstream evaluation metrics, i.e., reference-based metrics, introduce bias into the multi-reference evaluation because they extract edits without considering the presence of multiple references. To overcome the problem, we propose Chunk-LEvel Multi-reference Evaluation (CLEME) designed to evaluate GEC systems in multi-reference settings. First, CLEME builds chunk sequences with consistent boundaries for the source, the hypothesis and all the references, thus eliminating the bias caused by inconsistent edit boundaries. Then, based on the discovery that there exist boundaries between different grammatical errors, we automatically determine the grammatical error boundaries and compute F$_{0.5}$ scores in a novel way. Our proposed CLEME approach consistently and substantially outperforms existing reference-based GEC metrics on multiple reference sets in both corpus-level and sentence-level settings. Extensive experiments and detailed analyses demonstrate the correctness of our discovery and the effectiveness of our designed evaluation metric.Keywords
All Related Versions
This publication has 21 references indexed in Scilit:
- Grammatical Error Correction: A Survey of the State of the ArtComputational Linguistics, 2023
- A Survey of Natural Language GenerationACM Computing Surveys, 2022
- Czech Grammar Error Correction with a Large and Diverse CorpusTransactions of the Association for Computational Linguistics, 2022
- Revisiting Grammatical Error Correction Evaluation and Beyond2022
- On the Limitations of Reference-Free Evaluations of Generated Text2022
- Interpretability for Language Learners Using Example-Based Grammatical Error Correction2022
- The Past Mistake is the Future Wisdom: Error-driven Contrastive Probability Optimization for Chinese Spell Checking2022
- Taking the Correction Difficulty into Account in Grammatical Error Correction Evaluation2020
- Enabling Robust Grammatical Error Correction in New Domains: Data Sets, Metrics, and AnalysesTransactions of the Association for Computational Linguistics, 2019
- Reassessing the Goals of Grammatical Error Correction: Fluency Instead of GrammaticalityTransactions of the Association for Computational Linguistics, 2016