CLSE: Corpus of Linguistically Significant Entities
Authors:
- Aleksandr ChuklinACAleksandr Chuklin
- Justin ZhaoJZJustin Zhao
- Mihir KaleMKMihir Kale
Preprint
Abstract
One of the biggest challenges of natural language generation (NLG) is the proper handling of named entities. Named entities are a common source of grammar mistakes such as wrong prepositions, wrong article handling, or incorrect entity inflection. Without factoring linguistic representation, such errors are often underrepresented when evaluating on a small set of arbitrarily picked argument values, or when translating a dataset from a linguistically simpler language, like English, to a linguistically complex language, like Russian. However, for some applications, broadly precise grammatical correctness is critical -- native speakers may find entity-related grammar errors silly, jarring, or even offensive. To enable the creation of more linguistically diverse NLG datasets, we release a Corpus of Linguistically Significant Entities (CLSE) annotated by linguist experts. The corpus includes 34 languages and covers 74 different semantic types to support various applications from airline ticketing to video games. To demonstrate one possible use of CLSE, we produce an augmented version of the Schema-Guided Dialog Dataset, SGD-CLSE. Using the CLSE's entities and a small number of human translations, we create a linguistically representative NLG evaluation benchmark in three languages: French (high-resource), Marathi (low-resource), and Russian (highly inflected language). We establish quality baselines for neural, template-based, and hybrid NLG systems and discuss the strengths and weaknesses of each approach.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine TranslationTransactions of the Association for Computational Linguistics, 2021
- Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questions2021
- A Simple Recipe for Multilingual Grammatical Error Correction2021
- Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics2021
- Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue DatasetProceedings of the AAAI Conference on Artificial Intelligence, 2020
- Machine Translation Pre-training for Data-to-Text Generation - A Case Study in Czech2020
- Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation2020
- BLEURT: Learning Robust Metrics for Text Generation2020
- Text Summarization Techniques: A Brief SurveyInternational Journal of Advanced Computer Science and Applications, 2017
- Building applied natural language generation systemsNatural Language Engineering, 1997