Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions
Authors:
Top Cited Papers
- 1 June 2020
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 7887-7896
Abstract
Generative convolutional deep neural networks, e.g. popular GAN architectures, are relying on convolution based up-sampling methods to produce non-scalar outputs like images or video sequences. In this paper, we show that common up-sampling methods, i.e. known as up-convolution or transposed convolution, are causing the inability of such models to reproduce spectral distributions of natural training data correctly. This effect is independent of the underlying architecture and we show that it can be used to easily detect generated data like deepfakes with up to 100% accuracy on public benchmarks. To overcome this drawback of current generative models, we propose to add a novel spectral regularization term to the training optimization objective. We show that this approach not only allows to train spectral consistent GANs that are avoiding high frequency errors. Also, we show that a correct approximation of the frequency spectrum has positive effects on the training stability and output quality of generative networks.Keywords
All Related Versions
This publication has 26 references indexed in Scilit:
- Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks2017
- Globally and locally consistent image completionACM Transactions on Graphics, 2017
- Semantic Image Inpainting with Deep Generative Models2017
- Generative Face Completion2017
- DeLiGAN: Generative Adversarial Networks for Diverse and Limited Data2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Context Encoders: Feature Learning by Inpainting2016
- Rethinking the Inception Architecture for Computer Vision2016
- Deep Learning Face Attributes in the Wild2015
- An Introduction to Harmonic Analysis2004