Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors
Authors:
- Wenqiang ChenWCWenqiang Chen
- Jiaxuan ChengJCJiaxuan Cheng
- Leyao WangLWLeyao Wang
- Wei ZhaoWZWei Zhao
- Wojciech MatusikWMWojciech Matusik
Preprint
Abstract
Visual Question-Answering, a technology that generates textual responses from an image and natural language question, has progressed significantly. Notably, it can aid in tracking and inquiring about daily activities, crucial in healthcare monitoring, especially for elderly patients or those with memory disabilities. However, video poses privacy concerns and has a limited field of view. This paper presents Sensor2Text, a model proficient in tracking daily activities and engaging in conversations using wearable sensors. The approach outlined here tackles several challenges, including low information density in wearable sensor data, insufficiency of single wearable sensors in human activities recognition, and model's limited capacity for Question-Answering and interactive conversations. To resolve these obstacles, transfer learning and student-teacher networks are utilized to leverage knowledge from visual-language models. Additionally, an encoder-decoder neural network model is devised to jointly process language and sensor data for conversational purposes. Furthermore, Large Language Models are also utilized to enable interactive capabilities. The model showcases the ability to identify human activities and engage in Q\&A dialogues using various wearable sensor modalities. It performs comparably to or better than existing visual-language models in both captioning and conversational tasks. To our knowledge, this represents the first model capable of conversing about wearable sensor data, offering an innovative approach to daily activity tracking that addresses privacy and field-of-view limitations associated with current vision-based solutions.Keywords
All Related Versions
This publication has 32 references indexed in Scilit:
- Improved Baselines with Visual Instruction Tuning2024
- ImageBind One Embedding Space to Bind Them All2023
- Diverse Video Captioning by Adaptive Spatio-temporal Attention2022
- Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learningScientific Reports, 2021
- In-Home Daily-Life Captioning Using Radio Signals2020
- MMAct: A Large-Scale Dataset for Cross Modal Human Action Understanding2019
- Making the Invisible Visible: Action Recognition Through Walls and Occlusions2019
- A Survey of Privacy Concerns in Wearable Devices2018
- SPICE: Semantic Propositional Image Caption Evaluation2016
- Relation between object properties and EMG during reaching to graspJournal of Electromyography and Kinesiology, 2013