Learning an Interpretable Stylized Subspace for 3D-Aware Animatable Artforms
- 9 February 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Visualization and Computer Graphics
- Vol. 31 (2) , 1465-1477
Abstract
Throughout history, static paintings have captivated viewers within display frames, yet the possibility of making these masterpieces vividly interactive remains intriguing. This research paper introduces 3DArtmator, a novel approach that aims to represent artforms in a highly interpretable stylized space, enabling 3D-aware animatable reconstruction and editing. Our rationale is to transfer the interpretability and 3D controllability of the latent space in a 3D-aware GAN to a stylized sub-space of a customized GAN, revitalizing the original artforms. To this end, the proposed two-stage optimization framework of 3DArtmator begins with discovering an anchor in the original latent space that accurately mimics the pose and content of a given art painting. This anchor serves as a reliable indicator of the original latent space local structure, therefore sharing the same editable predefined expression vectors. In the second stage, we train a customized 3D-aware GAN specific to the input artform, while enforcing the preservation of the original latent local structure through a meticulous style-directional difference loss. This approach ensures the creation of a stylized sub-space that remains interpretable and retains 3D control. The effectiveness and versatility of 3DArtmator are validated through extensive experiments across a diverse range of art styles. With the ability to generate 3D reconstruction and editing for artforms while maintaining interpretability, 3DArtmator opens up new possibilities for artistic exploration and engagement.Keywords
Funding Information
- Guangdong International Technology Cooperation Project (2022A0505050009)
- China National Key R&D Program (2023YFE0202700)
- Key-Area Research and Development Program of Guangzhou City (2023B01J0022)
- Guangdong Natural Science Funds for Distinguished Young Scholar (2023B1515020097)
- Singapore MOE Tier 1 Funds (MSS23C002)
- National Research Foundation Singapore under the AI Singapore Programme (AISG3-GV-2023-011)
This publication has 50 references indexed in Scilit:
- Generative Multiplane Images: Making a 2D GAN 3D-Aware2022
- Unified Implicit Neural Stylization2022
- Editing Out-of-Domain GAN Inversion via Differential Activations2022
- ARF: Artistic Radiance Fields2022
- Vector Quantized Image-to-Image Translation2022
- A Style-Based Generator Architecture for Generative Adversarial Networks2019
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018
- StyleBank: An Explicit Representation for Neural Image Style Transfer2017
- Deep Photo Style Transfer2017
- Image Style Transfer Using Convolutional Neural Networks2016