Neural Canvas: Supporting Scenic Design Prototyping by Integrating 3D Sketching and Generative AI
Open Access
- 11 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-18
Abstract
We propose Neural Canvas, a lightweight 3D platform that integrates sketching and a collection of generative AI models to facilitate scenic design prototyping. Compared with traditional 3D tools, sketching in a 3D environment helps designers quickly express spatial ideas, but it does not facilitate the rapid prototyping of scene appearance or atmosphere. Neural Canvas integrates generative AI models into a 3D sketching interface and incorporates four types of projection operations to facilitate 2D-to-3D content creation. Our user study shows that Neural Canvas is an effective creativity support tool, enabling users to rapidly explore visual ideas and iterate 3D scenic designs. It also expedites the creative process for both novices and artists who wish to leverage generative AI technology, resulting in attractive and detailed 3D designs created more efficiently than using traditional modeling tools or individual generative AI platforms.Keywords
Funding Information
- Guangzhou-HKUST(GZ) Joint Funding (#2023A03J0670)
This publication has 56 references indexed in Scilit:
- Generative Multiplane Images: Making a 2D GAN 3D-Aware2022
- Multimodal Conditional Image Synthesis with Product-of-Experts GANs2022
- Deep learning for monocular depth estimation: A reviewNeurocomputing, 2021
- Array programming with NumPyNature, 2020
- Semantic Understanding of Scenes Through the ADE20K DatasetInternational Journal of Computer Vision, 2018
- Sketch-based modeling: A surveyComputers & Graphics, 2009
- An Empirical Evaluation of the System Usability ScaleInternational Journal of Human–Computer Interaction, 2008
- Modeling the World from Internet Photo CollectionsInternational Journal of Computer Vision, 2007
- The Necessity for Drawing2005
- Stimulus information as a determinant of reaction time.Journal of Experimental Psychology, 1953