Reassessing Multimodal Pathways for Learning Action Meaning
Preprint
- 22 December 2025
- preprint
- Published by MDPI AG in Preprints.org
Abstract
The semantic interpretation of actions is deeply intertwined with how change unfolds over time, space, and interaction. Prior theoretical and computational work has suggested that explicitly modeling three-dimensional motion---including object positions and orientations evolving through time---should offer a privileged pathway for encoding fine-grained verb meaning, especially for distinctions such as \textit{roll} versus \textit{slide}. At the same time, the vast majority of multimodal language models rely almost exclusively on two-dimensional visual inputs, implicitly assuming that such projections suffice to ground linguistic meaning. In this work, we revisit this assumption through a systematic and tightly controlled comparison of visual and motion-based modalities. We construct self-supervised encoders over both 2D video observations and 3D trajectory data, and probe the resulting representations for their capacity to discriminate verb-level semantic categories. Contrary to prevailing intuition, our empirical analysis reveals that representations learned from 2D visual streams are competitive with, and in some cases indistinguishable from, those derived from explicit 3D trajectories. These findings complicate the widely held belief that richer environmental encodings automatically lead to superior semantic representations, and suggest that the relationship between perceptual fidelity and linguistic abstraction is more nuanced than often assumed. Our study offers early evidence that effective verb representation may emerge from multiple perceptual pathways, motivating a rethinking of how embodiment and modality interact in multimodal language learning.Keywords
All Related Versions
This publication has 27 references indexed in Scilit:
- Grammar induction from visual, speech and textArtificial Intelligence, 2025
- Knowledge-enhanced event relation extraction via event ontology promptInformation Fusion, 2023
- Nonautoregressive Encoder–Decoder Neural Framework for End-to-End Aspect-Based Sentiment Triplet ExtractionIEEE Transactions on Neural Networks and Learning Systems, 2021
- A span-graph neural model for overlapping entity relation extraction in biomedical textsBioinformatics, 2020
- Enriching contextualized language model from knowledge graph for biomedical information extractionBriefings in Bioinformatics, 2020
- Dispatched attention with multi-task learning for nested mention recognitionInformation Sciences, 2020
- A tree-based neural network model for biomedical event trigger detectionInformation Sciences, 2020
- Deep Visual-Semantic Alignments for Generating Image DescriptionsIEEE Transactions on Pattern Analysis and Machine Intelligence, 2016
- Deep learningNature, 2015
- Twenty Years of Mixture of ExpertsIEEE Transactions on Neural Networks and Learning Systems, 2012