Rewriting geometric rules of a GAN
Open Access
- 22 July 2022
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Graphics
- Vol. 41 (4) , 1-16
Abstract
Deep generative models make visual content creation more accessible to novice users by automating the synthesis of diverse, realistic content based on a collected dataset. However, the current machine learning approaches miss a key element of the creative process - the ability to synthesize things that go far beyond the data distribution and everyday experience. To begin to address this issue, we enable a user to "warp" a given model by editing just a handful of original model outputs with desired geometric changes. Our method applies a low-rank update to a single model layer to reconstruct edited examples. Furthermore, to combat overfitting, we propose a latent space augmentation method based on style-mixing. Our method allows a user to create a model that synthesizes endless objects with defined geometric changes, enabling the creation of a new generative model without the burden of curating a large-scale dataset. We also demonstrate that edited models can be composed to achieve aggregated effects, and we present an interactive interface to enable users to create new models through composition. Empirical measurements on multiple test cases suggest the advantage of our method against recent GAN fine-tuning methods. Finally, we showcase several applications using the edited models, including latent space interpolation and image editing.Keywords
All Related Versions
This publication has 29 references indexed in Scilit:
- Visual attribute transfer through deep image analogyACM Transactions on Graphics, 2017
- Image-to-Image Translation with Conditional Adversarial Networks2017
- Generative Visual Manipulation on the Natural Image Manifold2016
- Data-driven hallucination of different times of day from a single outdoor photoACM Transactions on Graphics, 2013
- PatchMatch2009
- Automatic Panoramic Image Stitching using Invariant FeaturesInternational Journal of Computer Vision, 2006
- Image deformation using moving least squaresACM Transactions on Graphics, 2006
- Image quality assessment: from error visibility to structural similarityIEEE Transactions on Image Processing, 2004
- As-rigid-as-possible shape interpolation2000
- Random sample consensusCommunications of the ACM, 1981