Knowledge Distillation for Single Image Super-Resolution via Contrastive Learning
Open Access
- 30 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1079-1083
Abstract
In recent years, thanks to the vigorous development of deep learning, single image super-resolution has advanced greatly. Most super-resolution (SR) methods use convolution layers to construct the network, which achieves superior results over the traditional methods based on manual features. However, most methods based on convolutional neural networks (CNN) blindly deepen the depth of the network leading to a large number of model parameters, which inevitably brings huge computing overhead and memory consumption, and greatly limits the application in resource-limited devices. In order to alleviate this problem, a knowledge distillation framework based on contrastive learning is proposed to compress and accelerate the SR model with enormous parameters. The student network is directly constructed by reducing the number of layers of the teacher network. In particular, the proposed method distills the statistical information of the intermediate feature maps from the teacher network to train the lightweight student network. In addition, through explicit knowledge transfer, a novel contrastive loss is introduced to improve the reconstruction performance of the student network. Experiments show that the proposed contrastive distillation framework can effectively compress the model scale with an acceptable loss of performance.Keywords
Funding Information
- National Natural Science Foundation of China (62306064)
- Research Funding of Science and Technology on Information System Engineering Laboratory (6142101230202)
This publication has 16 references indexed in Scilit:
- Spherical Pseudo-Cylindrical Representation for Omnidirectional Image Super-resolutionProceedings of the AAAI Conference on Artificial Intelligence, 2024
- Class Attention Transfer Based Knowledge Distillation2023
- Multi-Mode Online Knowledge Distillation for Self-Supervised Visual Representation Learning2023
- State-Aware Compositional Learning Toward Unbiased Training for Scene Graph GenerationIEEE Transactions on Image Processing, 2022
- Towards Open-Vocabulary Scene Graph Generation with Prompt-Based Finetuning2022
- Image Super-Resolution Using Very Deep Residual Channel Attention Networks2018
- Residual Dense Network for Image Super-Resolution2018
- Unsupervised Feature Learning via Non-parametric Instance Discrimination2018
- Accurate Image Super-Resolution Using Very Deep Convolutional Networks2016
- Image Super-Resolution Using Deep Convolutional NetworksIEEE Transactions on Pattern Analysis and Machine Intelligence, 2015