Abstract
Natural language (NL) provides an intuitive interface for querying video datasets. However, existing approaches that use Multimodal Large Language Models to directly map NL queries to video results offer limited feedback mechanisms beyond prompt engineering. We present WayPoint, a prototype system for NL-based querying of spatio-temporal video events through a user-in-the-loop approach. WayPoint translates NL queries into a keyframe-inspired intermediate representation that enables fuzzy matching of spatio-temporal constraints. This structured representation allows users to refine query semantics through natural language feedback that is applied to the query logic and has deterministic effects on execution. Users can also tune retrieval performance by providing feedback on retrieved results. We demonstrate WayPoint's query synthesis and retrieval refinement workflows on the nuScenes dataset.
Funding Information
  • National Science Foundation (IIS-2335881)