Distilling Representations from GAN Generator via Squeeze and Span
Preprint
Abstract
In recent years, generative adversarial networks (GANs) have been an actively studied topic and shown to successfully produce high-quality realistic images in various domains. The controllable synthesis ability of GAN generators suggests that they maintain informative, disentangled, and explainable image representations, but leveraging and transferring their representations to downstream tasks is largely unexplored. In this paper, we propose to distill knowledge from GAN generators by squeezing and spanning their representations. We squeeze the generator features into representations that are invariant to semantic-preserving transformations through a network before they are distilled into the student network. We span the distilled representation of the synthetic domain to the real domain by also using real training data to remedy the mode collapse of GANs and boost the student network performance in a real domain. Experiments justify the efficacy of our method and reveal its great significance in self-supervised representation learning. Code is available at https://github.com/yangyu12/squeeze-and-span.Keywords
All Related Versions
This publication has 30 references indexed in Scilit:
- Contrastive Multiview Coding2020
- Analyzing and Improving the Image Quality of StyleGAN2020
- Interpreting the Latent Space of GANs for Semantic Face Editing2020
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- A Gift from Knowledge Distillation: Fast Optimization, Network Minimization and Transfer Learning2017
- Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles2016
- Learning Representations for Automatic Colorization2016
- Context Encoders: Feature Learning by Inpainting2016
- Deep Residual Learning for Image Recognition2016
- Dimensionality Reduction by Learning an Invariant Mapping2006