Dataset Distillation via a Noise-Unconstrained Generative Model
Top Cited Papers
- 6 May 2026
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-18
Abstract
Dataset distillation (DD) aims to synthesize a more compact dataset than the original one and models trained on it are expected to have the same generalization capabilities as on the original dataset. Previous work via a generative model (GM) faces several limitations. First, GM struggles to generate representative samples due to a lack of constraints. Second, it overlooks the relationships between generated samples, limiting its effectiveness. In this paper, a new noise-unconstrained GM-based DD framework is proposed. In the distillation stage, an adaptive matching coefficient is introduced to align generated images with representative class elements and the MiniMax loss function is extended to reduce the optimization difficulty. In the deployment stage, features among each generative image are ensembled by gradient-matching based DD. Theoretical analysis based on McDiarmid's inequality demonstrates that the proposed components can reduce the generalization error of the original baseline method. We also provide insights into the potential of generated images as an effective proxy dataset for DD. For example, on the ImageWoof dataset with 50 distilled images per class using a 6-layer ConvNet for evaluation, generated images outperform 25%, 50%, and 75% original images by 8.4%, 6.3%, and 8.3% in distillation performance. Our method effectively handles both low- and high-resolution datasets, with experiments on 11 benchmarks demonstrating its efficacy.Keywords
Funding Information
- National Natural Science Foundation of China (T2525004)
- Shenzhen Medical Research Fund (E250200613)
- State Key Laboratory of Industrial Control Technology, China (ICT2026A25)
This publication has 30 references indexed in Scilit:
- DiM: Distilling Dataset into Generative Model2025
- Distill Gold from Massive Ores: Bi-level Data Pruning Towards Efficient Dataset Distillation2024
- Generalizing Dataset Distillation via Deep Generative Prior2023
- Improved Distribution Matching for Dataset Condensation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Dataset Distillation by Matching Training Trajectories2022
- CAFE: Learning to Condense Dataset by Aligning Features2022
- Contrastive Multiview Coding2020
- Fast interpolation-based t-SNE for improved visualization of single-cell RNA-seq dataNature Methods, 2019
- ImageNet: A large-scale hierarchical image database2009