Characterizing and Improving Stability in Neural Style Transfer
Preprint
Abstract
Recent progress in style transfer on images has focused on improving the quality of stylized images and speed of methods. However, real-time methods are highly unstable resulting in visible flickering when applied to videos. In this work we characterize the instability of these methods by examining the solution set of the style transfer objective. We show that the trace of the Gram matrix representing style is inversely related to the stability of the method. Then, we present a recurrent convolutional network for real-time video style transfer which incorporates a temporal consistency loss and overcomes the instability of prior methods. Our networks can be applied at any resolution, do not re- quire optical flow at test time, and produce high quality, temporally consistent stylized videos in real-time.Keywords
All Related Versions
This publication has 14 references indexed in Scilit:
- Controlling Perceptual Factors in Neural Style Transfer2017
- FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks2017
- Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis2017
- A Quantitative Analysis of Current Practices in Optical Flow Estimation and the Principles Behind ThemInternational Journal of Computer Vision, 2013
- Lessons and Insights from Creating a Synthetic Optical Flow Benchmark2012
- A Parametric Texture Model Based on Joint Statistics of Complex Wavelet CoefficientsInternational Journal of Computer Vision, 2000
- Exploring texture ensembles by efficient Markov chain Monte Carlo-Toward a "trichromacy" theory of textureIEEE Transactions on Pattern Analysis and Machine Intelligence, 2000
- Performance of optical flow techniquesInternational Journal of Computer Vision, 1994
- On the computation of motion from sequences of images-A reviewProceedings of the IEEE, 1988
- Determining optical flowArtificial Intelligence, 1981