Large-scale Dataset Pruning with Dynamic Uncertainty
Authors:
- 17 June 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7713-7722
Abstract
The state of the art of many learning tasks, e.g., image classification, is advanced by collecting larger datasets and then training larger models on them. As the outcome, the increasing computational cost is becoming unaffordable. In this paper, we investigate how to prune the large-scale datasets, and thus produce an informative subset for training sophisticated deep models with negligible performance drop. We propose a simple yet effective dataset pruning method by exploring both the prediction uncertainty and training dynamics. We study dataset pruning by measuring the variation of predictions during the whole training process on large-scale datasets, i.e., ImageNet-1K and ImageNet-21K, and advanced models, i.e., Swin Transformer and ConvNeXt. Extensive experimental results indicate that our method outperforms the state of the art and achieves 25% lossless pruning ratio on both ImageNet1K and ImageNet-21K. The code and pruned datasets are available at https://github.com/BAAI-DCAI/Dataset-Pruning.Keywords
All Related Versions
Funding Information
- National Science and Technology Major Project
This publication has 18 references indexed in Scilit:
- EVA: Exploring the Limits of Masked Visual Representation Learning at Scale2023
- DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning2022
- Dataset Distillation by Matching Training Trajectories2022
- A ConvNet for the 2020s2022
- SimMIM: a Simple Framework for Masked Image Modeling2022
- CAFE: Learning to Condense Dataset by Aligning Features2022
- Natural Adversarial Examples2021
- Core‐sets: An updated surveyWIREs Data Mining and Knowledge Discovery, 2019
- Deep Residual Learning for Image Recognition2016
- ImageNet: A large-scale hierarchical image database2009