Make Every Example Count: On Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets
Preprint
Abstract
Increasingly larger datasets have become a standard ingredient to advancing the state of the art in NLP. However, data quality might have already become the bottleneck to unlock further gains. Given the diversity and the sizes of modern datasets, standard data filtering is not straight-forward to apply, because of the multifacetedness of the harmful data and elusiveness of filtering rules that would generalize across multiple tasks. We study the fitness of task-agnostic self-influence scores of training examples for data cleaning, analyze their efficacy in capturing naturally occurring outliers, and investigate to what extent self-influence based data cleaning can improve downstream performance in machine translation, question answering and text classification, building up on recent approaches to self-influence calculation and automated curriculum learning.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading ComprehensionACM Computing Surveys, 2023
- Scaling Up Influence FunctionsProceedings of the AAAI Conference on Artificial Intelligence, 2022
- Understanding deep learning (still) requires rethinking generalizationCommunications of the ACM, 2021
- Explaining Black Box Predictions and Unveiling Data Artifacts through Influence Functions2020
- ParaCrawl: Web-Scale Acquisition of Parallel Corpora2020
- Natural Questions: A Benchmark for Question Answering ResearchTransactions of the Association for Computational Linguistics, 2019
- Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification2019
- Classification in the Presence of Label Noise: A SurveyIEEE Transactions on Neural Networks and Learning Systems, 2013
- Curriculum learning2009
- Finite-time Analysis of the Multiarmed Bandit ProblemMachine Learning, 2002