Image Distillation with the Machine-Learned Gradient of the Loss Function and the K-Means Method
Open Access
- 25 November 2025
- journal article
- research article
- Published by MDPI AG in Mathematics
- Vol. 13 (23) , 3785
Abstract
Image distillation is becoming a hot area of machine learning (ML) research because it minimizes the computer resources used for training of convolutional neural networks (CNNs). In this paper, we propose a novel method for image distillation that learns the gradient of the loss function after training a CNN with the original training image. It then modifies the images using the learned gradient and applies K-means clustering to split the modified images into clusters, in a class-wise fashion, and select their centroids as distilled images. We evaluated the capabilities of our method on MNIST, Fashion-MNIST, CIFAR-10, and the Oral Cancer Image Database (OCID) using three CNN models (Baseline, ResNet, and LeNet). The results show that when training the classifiers (CNN) with distilled images, we obtained accuracies of 99.93% for MNIST, 99.46% for Fashion-MNIST, 99.05% for CIFAR-10, and 91.8% for OCID, which are comparable with the results after training with the entire training set. Comparison of our distillation method with existing contemporary methods such as KIP and RTP showed the supremacy of the former even if trained with fewer epochs. A bottleneck of our method is that it works with gradient-based optimizers.Keywords
This publication has 19 references indexed in Scilit:
- Automated identification of ground kinematic patterns based on InSAR time series displacement and K-SC clusteringEngineering Geology, 2025
- A comprehensive dataset of annotated oral cavity images for diagnosis of oral cancer and oral potentially malignant disordersOral Oncology, 2024
- Adaptive selection of local and non-local attention mechanisms for speech enhancementNeural Networks, 2024
- Dataset Distillation: A Comprehensive ReviewIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- A Comprehensive Survey of Dataset DistillationIEEE Transactions on Pattern Analysis and Machine Intelligence, 2023
- K-means clustering algorithms: A comprehensive review, variants analysis, and advances in the era of big dataInformation Sciences, 2022
- Understanding of a convolutional neural network2017
- Two-dimensional pca: a new approach to appearance-based face representation and recognitionIEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
- From few to many: illumination cone models for face recognition under variable lighting and poseIEEE Transactions on Pattern Analysis and Machine Intelligence, 2001
- Gradient-based learning applied to document recognitionProceedings of the IEEE, 1998