LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
Top Cited Papers
Open Access
- 11 May 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1-22
Abstract
We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR’s cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.Keywords
All Related Versions
This publication has 25 references indexed in Scilit:
- Adding Conditional Control to Text-to-Image Diffusion Models2023
- PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning2023
- Magic3D: High-Resolution Text-to-3D Content Creation2023
- HOLODIFFUSION: Training a 3D Diffusion Model Using 2D Images2023
- HexPlane: A Fast Representation for Dynamic Scenes2023
- Learning Transferable Human-Object Interaction Detector with Natural Language Supervision2022
- Interactive example-based terrain authoring with conditional generative adversarial networksACM Transactions on Graphics, 2017
- DeepSpace: Mood-Based Image Texture Generation for Virtual Reality from Music2017
- Procedurally generated virtual reality from 3D reconstructed physical space2016
- A Survey on Procedural Modelling for Virtual WorldsComputer Graphics Forum, 2014