Estimating Example Difficulty Using Variance of Gradients
Preprint
Abstract
In machine learning, a question of great interest is understanding what examples are challenging for a model to classify. Identifying atypical examples ensures the safe deployment of models, isolates samples that require further human inspection and provides interpretability into model behavior. In this work, we propose Variance of Gradients (VoG) as a valuable and efficient metric to rank data by difficulty and to surface a tractable subset of the most challenging examples for human-in-the-loop auditing. We show that data points with high VoG scores are far more difficult for the model to learn and over-index on corrupted or memorized examples. Further, restricting the evaluation to the test set instances with the lowest VoG improves the model's generalization performance. Finally, we show that VoG is a valuable and efficient ranking for out-of-distribution detection.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Does your dermatology classifier know what it doesn’t know? Detecting the long-tail of unseen conditionsMedical Image Analysis, 2022
- Natural Adversarial Examples2021
- Moving beyond “algorithmic bias is a data problem”Patterns, 2021
- Deep learning for financial applications : A surveyApplied Soft Computing, 2020
- Hidden stratification causes clinically meaningful failures in machine learning for medical imaging2020
- Automated pulmonary nodule detection in CT images using deep convolutional neural networksPattern Recognition, 2019
- 3D deep learning for detecting pulmonary nodules in CT scansJournal of the American Medical Informatics Association, 2018
- Deep Residual Learning for Image Recognition2016
- Case-Based Explanation for Artificial Neural Nets2000
- LIII. On lines and planes of closest fit to systems of points in spaceJournal of Computers in Education, 1901