MoViS: Interactive 3D Human Motion Authoring via Zero-3D Visual Prompts
Open Access
- 9 November 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 9363-9372
Abstract
Generating spatially-aware 3D human motions from natural language remains a fundamental challenge, as text inherently lacks precise trajectory control. While recent methods condition generation on explicit 3D scenes to enforce physical grounding, they impose a severe human-computer interaction (HCI) bottleneck, demanding specialized 3D expertise and cumbersome user inputs. To democratize spatial control, we propose a paradigm shift: an interactive, zero-3D multimodal authoring framework named MoViS. MoViS empowers users to dictate complex spatial intent simply by drawing a 2D planar sketch (Visual Prompt) over a scene image, complemented by a text instruction. However, inferring 3D physical geometry from ambiguous 2D sketches presents extreme challenges in perspective scale and long-horizon intent drift. To overcome these, MoViS introduces a Physics-Grounded Cross-Modal Alignment mechanism, uniting 3D landmark distillation and continuous 2D perspective projection to embed true spatial awareness into a discrete generative backbone. Furthermore, we propose a cognitive-inspired Sketch-Driven Compositional Decoding strategy to robustly execute complex, multi-stage tasks. To catalyze this new paradigm, we construct IndoorViS, a large-scale multimodal dataset featuring 2,015 daily tasks across 1,173 diverse scenes. Extensive experiments demonstrate that MoViS strictly maintains the physical fidelity of explicit 3D-aware models while reducing human authoring overhead by over 90%, establishing a new standard for accessible and expressive embodied agent control.Keywords
This publication has 32 references indexed in Scilit:
- Temporal consistency-aware text-to-motion generationVisual Intelligence, 2026
- A Survey and Framework for Education in the MetaverseIEEE Access, 2025
- Generating Human Motion from Textual Descriptions with Discrete Representations2023
- TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts2022
- Generating Diverse and Natural 3D Human Motions from Text2022
- Learning to generate line drawings that convey geometry and semantics2022
- Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes2021
- Expressive Body Capture: 3D Hands, Face, and Body From a Single Image2019
- VirtualHome: Simulating Household Activities Via Programs2018
- The KIT Motion-Language DatasetBig Data, 2016