Efficient Online Scalar Annotation with Bounded Support
Preprint
Abstract
We describe a novel method for efficiently eliciting scalar annotations for dataset construction and system quality estimation by human judgments. We contrast direct assessment (annotators assign scores to items directly), online pairwise ranking aggregation (scores derive from annotator comparison of items), and a hybrid approach (EASL: Efficient Annotation of Scalar Labels) proposed here. Our proposal leads to increased correlation with ground truth, at far greater annotator efficiency, suggesting this strategy as an improved mechanism for dataset creation and manual system evaluation.Keywords
All Related Versions
This publication has 5 references indexed in Scilit:
- RankME: Reliable Human Ratings for Natural Language Generation2018
- Ordinal Common-sense InferenceTransactions of the Association for Computational Linguistics, 2017
- Reassessing the Goals of Grammatical Error Correction: Fluency Instead of GrammaticalityTransactions of the Association for Computational Linguistics, 2016
- Findings of the 2014 Workshop on Statistical Machine Translation2014
- Judgment under Uncertainty: Heuristics and BiasesScience, 1974