Speak to the City: Multimodal Resolution for Outside-the-Vehicle References
Open Access
- 19 September 2026
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 95-105
Abstract
As autonomous vehicles and Extended Reality (XR) headsets enable novel in-car interactions, seamlessly querying physical landmarks, known as Outside-the-Vehicle Referencing (OVR), remains challenging due to ego-motion and referential ambiguity. We present a robust, multimodal OVR framework fusing user gaze and natural language to identify Points of Interest (POIs). To address the scarcity of dynamic vehicular data, we developed a VR-based pipeline synchronizing 360-degree transit videos with vehicle GNSS telemetry. Through a user study (N=46) mapping passenger head orientation into a 3D geospatial Digital Twin, we captured authentic gaze-speech behaviors. We subsequently trained a lightweight Transformer network, leveraging LLMs to dynamically align continuous spatial gaze vectors with discrete verbal context. Experimental results demonstrate high accuracy and low computational overhead, achieving an 83.33% Top-1 accuracy (87.72% Top-2) and an average inference time of 24.3 milliseconds. This real-time paradigm effectively resolves referential ambiguity, enabling context-aware spatial retrieval for passengers within the vehicle.Keywords
All Related Versions
This publication has 18 references indexed in Scilit:
- Blending the Worlds: An evaluation of World-Fixed Visual Appearances in Automotive Augmented Reality2025
- Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers2024
- It’s all about you: Personalized in-Vehicle Gesture Recognition with a Time-of-Flight Camera2023
- ML-PersRef: A Machine Learning-based Personalized Multimodal Fusion Approach for Referencing Outside Objects From a Moving Vehicle2021
- Designing for a Convenient In-Car Passenger Experience: A Repertory Grid Study2021
- Multimodal Fusion Using Deep Learning Applied to Driver's Referencing of Outside-Vehicle Objects2021
- You Have a Point There2020
- I See Your Point2018
- How Does Eye-Gaze Relate to Gesture Movement in an Automotive Pointing Task?2017
- What the Eyes Say About SpeakingPsychological Science, 2000