Demonstration of W ay P oint : Interactive Natural Language Querying for Spatio-Temporal Video Events
Open Access
- 30 May 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 58-61
Abstract
Natural language (NL) provides an intuitive interface for querying video datasets. However, existing approaches that use Multimodal Large Language Models to directly map NL queries to video results offer limited feedback mechanisms beyond prompt engineering. We present WayPoint, a prototype system for NL-based querying of spatio-temporal video events through a user-in-the-loop approach. WayPoint translates NL queries into a keyframe-inspired intermediate representation that enables fuzzy matching of spatio-temporal constraints. This structured representation allows users to refine query semantics through natural language feedback that is applied to the query logic and has deterministic effects on execution. Users can also tune retrieval performance by providing feedback on retrieved results. We demonstrate WayPoint's query synthesis and retrieval refinement workflows on the nuScenes dataset.Keywords
Funding Information
- National Science Foundation (IIS-2335881)
This publication has 5 references indexed in Scilit:
- Self-Enhancing Video Data Management System for Compositional Events with Large Language ModelsProceedings of the ACM on Management of Data, 2025
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces2025
- A survey on multimodal large language modelsNational Science Review, 2024
- SketchQL: Video Moment Querying with a Visual Query InterfaceProceedings of the ACM on Management of Data, 2024
- Spatial and temporal constrained ranked retrieval over videosProceedings of the VLDB Endowment, 2022