VideoMat: Extracting PBR Materials from Video Diffusion Models
- 24 July 2025
- journal article
- Published by Wiley in Computer Graphics Forum
- Vol. 44 (4)
Abstract
We leverage finetuned video diffusion models, intrinsic decomposition of videos, and physically‐based differentiable rendering to generate high quality materials for 3D models given a text prompt or a single image. We condition a video diffusion model to respect the input geometry and lighting condition. This model produces multiple views of a given 3D model with coherent material properties. Secondly, we use a recent model to extract intrinsics (base color, roughness, metallic) from the generated video. Finally, we use the intrinsics alongside the generated video in a differentiable path tracer to robustly extract PBR materials directly compatible with common content creation tools.Keywords
This publication has 33 references indexed in Scilit:
- FlashTex: Fast Relightable Mesh Texturing with LightControlNet2024
- SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image Using Latent Video Diffusion2024
- TextureDreamer: Image-Guided Texture Synthesis through Geometry-Aware Diffusion2024
- Paint-it: Text-to-Texture Synthesis via Deep Convolutional Texture Map Optimization and Physically-Based Rendering2024
- Paint3D: Paint Anything 3D With Lighting-Less Texture Diffusion Models2024
- Objaverse: A Universe of Annotated 3D Objects2023
- Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models2023
- ABO: Dataset and Benchmarks for Real-World 3D Object Understanding2022
- Extracting Triangular 3D Models, Materials, and Lighting From Images2022
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric2018