SeMv-3D: Towards Semantic and Mutil-view Consistency simultaneously for General Text-to-3D Generation with Triplane Priors
Authors:
Preprint
Abstract
Recent advancements in generic 3D content generation from text prompts have been remarkable by fine-tuning text-to-image diffusion (T2I) models or employing these T2I models as priors to learn a general text-to-3D model. While fine-tuning-based methods ensure great alignment between text and generated views, i.e., semantic consistency, their ability to achieve multi-view consistency is hampered by the absence of 3D constraints, even in limited view. In contrast, prior-based methods focus on regressing 3D shapes with any view that maintains uniformity and coherence across views, i.e., multi-view consistency, but such approaches inevitably compromise visual-textual alignment, leading to a loss of semantic details in the generated objects. To achieve semantic and multi-view consistency simultaneously, we propose SeMv-3D, a novel framework for general text-to-3d generation. Specifically, we propose a Triplane Prior Learner (TPL) that learns triplane priors with 3D spatial features to maintain consistency among different views at the 3D level, e.g., geometry and texture. Moreover, we design a Semantic-aligned View Synthesizer (SVS) that preserves the alignment between 3D spatial features and textual semantics in latent space. In SVS, we devise a simple yet effective batch sampling and rendering strategy that can generate arbitrary views in a single feed-forward inference. Extensive experiments present our SeMv-3D's superiority over state-of-the-art performances with semantic and multi-view consistency in any view. Our code and more visual results are available at https://anonymous.4open.science/r/SeMv-3D-6425.Keywords
All Related Versions
This publication has 13 references indexed in Scilit:
- VolumeDiffusion: Feed-forward text-to-3D generation with efficient volumetric encoderGraphical Models, 2025
- RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D2024
- TextMesh: Generation of Realistic 3D Meshes From Text Prompts2024
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- Objaverse: A Universe of Annotated 3D Objects2023
- SDFusion: Multimodal 3D Shape Completion, Reconstruction, and Generation2023
- High-Resolution Image Synthesis with Latent Diffusion Models2022
- Efficient Geometry-aware 3D Generative Adversarial Networks2022
- Emerging Properties in Self-Supervised Vision Transformers2021
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018