A Human Intention and Motion Prediction Framework for Applications in Human-Centric Digital Twins
Open Access
- 1 October 2025
- journal article
- research article
- Published by MDPI AG in Biomimetics
- Vol. 10 (10) , 656
Abstract
In manufacturing settings where humans and machines collaborate, understanding and predicting human intention is crucial for enabling the seamless execution of tasks. This knowledge is the basis for creating an intelligent, symbiotic, and collaborative environment. However, current foundation models often fall short in directly anticipating complex tasks and producing contextually appropriate motion. This paper proposes a modular framework that investigates strategies for structuring task knowledge and engineering context-rich prompts to guide Vision–Language Models in understanding and predicting human intention in semi-structured environments. Our evaluation, conducted across three use cases of varying complexity, reveals a critical tradeoff between prediction accuracy and latency. We demonstrate that a Rolling Context Window strategy, which uses a history of frames and the previously predicted state, achieves a strong balance of performance and efficiency. This approach significantly outperforms single-image inputs and computationally expensive in-context learning methods. Furthermore, incorporating egocentric video views yields a substantial 10.7% performance increase in complex tasks. For short-term motion forecasting, we show that the accuracy of joint position estimates is enhanced by using historical pose, gaze data, and in-context examples.Keywords
This publication has 58 references indexed in Scilit:
- Dynamic scenario-enhanced diverse human motion prediction network for proactive human–robot collaboration in customized assembly tasksJournal of Intelligent Manufacturing, 2024
- Enhancing human–robot collaborative transportation through obstacle-aware vibrotactile warning and virtual fixturesRobotics and Autonomous Systems, 2024
- Embodied intelligence in manufacturing: leveraging large language models for autonomous industrial roboticsJournal of Intelligent Manufacturing, 2024
- ProgPrompt: program generation for situated robot task planning using large language modelsAutonomous Robots, 2023
- Deploying OWL ontologies for semantic mediation of mixed-reality interactions for human–robot collaborative assemblyJournal of Manufacturing Systems, 2023
- A method based on interpretable machine learning for recognizing the intensity of human engagement intentionScientific Reports, 2023
- On modelling and handling of flexible materials: A review on Digital Twins and planning systemsProcedia CIRP, 2021
- Digital twin of composite assembly manufacturing processInternational Journal of Production Research, 2020
- Development of An Augmented Reality Equipped Composites Bonded Assembly and Repair for Aerospace ApplicationsIFAC-PapersOnLine, 2020
- Theory of Mind for a Humanoid RobotAutonomous Robots, 2002