Estimating Example Difficulty using Variance of Gradients
Authors:
- 1 June 2022
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 10358-10368
Abstract
In machine learning, a question of great interest is understanding what examples are challenging for a model to classify. Identifying atypical examples ensures the safe de-ployment of models, isolates samples that require further human inspection and provides interpretability into model behavior. In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most chal-lenging examples for human-in-the-loop auditing. We show that data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or mem-orized examples. Further, restricting the evaluation to the test set instances with the lowest VoG improves the model's generalization performance. Finally, we show that VoG is a valuable and efficient ranking for out-of-distribution detection.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Prototype selection for interpretable classificationThe Annals of Applied Statistics, 2011
- Remarks on Some Nonparametric Estimates of a Density Function2011
- An introduction to ROC analysisPattern Recognition Letters, 2006
- Support Vector Data DescriptionMachine Learning, 2004
- Case-Based Explanation for Artificial Neural Nets2000
- Selecting Typical Instances in Instance-Based Learning1992
- The Detection of Errors in Multivariate Data Using Principal ComponentsJournal of the American Statistical Association, 1974
- On Estimation of a Probability Density Function and ModeThe Annals of Mathematical Statistics, 1962
- Remarks on Some Nonparametric Estimates of a Density FunctionThe Annals of Mathematical Statistics, 1956
- LIII. On lines and planes of closest fit to systems of points in spaceJournal of Computers in Education, 1901