Track, Inpaint, Resplat: Subject-driven 3D and 4D Generation with Progressive Texture Infilling
Preprint
Abstract
Current 3D/4D generation methods are usually optimized for photorealism, efficiency, and aesthetics. However, they often fail to preserve the semantic identity of the subject across different viewpoints. Adapting generation methods with one or few images of a specific subject (also known as Personalization or Subject-driven generation) allows generating visual content that align with the identity of the subject. However, personalized 3D/4D generation is still largely underexplored. In this work, we introduce TIRE (Track, Inpaint, REsplat), a novel method for subject-driven 3D/4D generation. It takes an initial 3D asset produced by an existing 3D generative model as input and uses video tracking to identify the regions that need to be modified. Then, we adopt a subject-driven 2D inpainting model for progressively infilling the identified regions. Finally, we resplat the modified 2D multi-view observations back to 3D while still maintaining consistency. Extensive experiments demonstrate that our approach significantly improves identity preservation in 3D/4D generation compared to state-of-the-art methods. Our project website is available at https://zsh2000.github.io/track-inpaint-resplat.github.io/.Keywords
All Related Versions
This publication has 70 references indexed in Scilit:
- DreamReward: Text-to-3D Generation with Human Preference2024
- GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation2024
- Make-Your-3D: Fast and Consistent Subject-Driven 3D Content Generation2024
- SC4D: Sparse-Controlled Video-to-4D Generation and Motion Transfer2024
- CRM: Single Image to 3D Textured Mesh with Convolutional Reconstruction Model2024
- STAG4D: Spatial-Temporal Anchored Generative 4D Gaussians2024
- Watch Your Steps: Local Image and Scene Editing by Text Instructions2024
- GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting2024
- ScaleDreamer: Scalable Text-to-3D Synthesis with Asynchronous Score Distillation2024
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis2020