Measuring Attribution in Natural Language Generation Models
Authors:
Open Access
- 1 December 2023
- journal article
- research article
- Published by MIT Press in Computational Linguistics
- Vol. 49 (4) , 777-840
Abstract
Large neural models have brought a new challenge to natural language generation (NLG): It has become imperative to ensure the safety and reliability of the output of models that generate freely. To this end, we present an evaluation framework, Attributable to Identified Sources (AIS), stipulating that NLG output pertaining to the external world is to be verified against an independent, provided source. We define AIS and a two-stage annotation pipeline for allowing annotators to evaluate model output according to annotation guidelines. We successfully validate this approach on generation datasets spanning three tasks (two conversational QA datasets, a summarization dataset, and a table-to-text dataset). We provide full annotation guidelines in the appendices and publicly release the annotated data at https://github.com/google-research-datasets/AIS.Keywords
All Related Versions
This publication has 31 references indexed in Scilit:
- Inherent Disagreements in Human Textual InferencesTransactions of the Association for Computational Linguistics, 2019
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Combining Fact Extraction and Verification with Neural Semantic Matching NetworksProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Dialogue Natural Language Inference2019
- QuAC: Question Answering in Context2018
- FEVER: a Large-scale Dataset for Fact Extraction and VERification2018
- Get To The Point: Summarization with Pointer-Generator Networks2017
- Challenges in Data-to-Document Generation2017
- Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond2016
- Why the Child's Theory of Mind Really Is a TheoryMind & Language, 1992