Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors
Open Access
- 21 November 2024
- journal article
- research article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 8 (4) , 1-26
Abstract
Visual Question-Answering, a technology that generates textual responses from an image and natural language question, has progressed significantly. Notably, it can aid in tracking and inquiring about daily activities, crucial in healthcare monitoring, especially for elderly patients or those with memory disabilities. However, video poses privacy concerns and has a limited field of view. This paper presents Sensor2Text, a model proficient in tracking daily activities and engaging in conversations using wearable sensors. The approach outlined here tackles several challenges, including low information density in wearable sensor data, insufficiency of single wearable sensors in human activities recognition, and model's limited capacity for Question-Answering and interactive conversations. To resolve these obstacles, transfer learning and student-teacher networks are utilized to leverage knowledge from visual-language models. Additionally, an encoder-decoder neural network model is devised to jointly process language and sensor data for conversational purposes. Furthermore, Large Language Models are also utilized to enable interactive capabilities. The model showcases the ability to identify human activities and engage in Q&A dialogues using various wearable sensor modalities. It performs comparably to or better than existing visual-language models in both captioning and conversational tasks. To our knowledge, this represents the first model capable of conversing about wearable sensor data, offering an innovative approach to daily activity tracking that addresses privacy and field-of-view limitations associated with current vision-based solutions.Keywords
All Related Versions
This publication has 38 references indexed in Scilit:
- Human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learningScientific Reports, 2021
- In-Home Daily-Life Captioning Using Radio Signals2020
- MMAct: A Large-Scale Dataset for Cross Modal Human Action Understanding2019
- G-Fall: Device-free and Training-free Fall Detection with Geophones2019
- FaceInput: A Hand-Free and Secure Text Entry System through Facial Vibration2019
- A Survey of Privacy Concerns in Wearable Devices2018
- ViType: A Cost Efficient On-Body Typing System through Vibration2018
- Sequence to Sequence -- Video to Text2015
- CIDEr: Consensus-based image description evaluation2015
- Relation between object properties and EMG during reaching to graspJournal of Electromyography and Kinesiology, 2013