Solving the Sensor-Based Activity Recognition Problem (SOAR): Self-Supervised, Multi-Modal Recognition of Activities from Wearable Sensors
Open Access
- 5 October 2024
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 1004-1007
Abstract
Feature extraction remains the core challenge in Human Activity Recognition (HAR) - the automated inference of activities being performed from sensor data. Over the past few years, the community has witnessed a shift from manual feature engineering using statistical metrics and distribution-based representations, to feature learning via neural networks. Particularly, self-supervised learning methods that leverage large-scale unlabeled data to train powerful feature extractors have gained significant traction, and various works have demonstrated their ability to train powerful feature extractors from large-scale unlabeled data. Recently, the advent of Large Language Models (LLMs) and multi-modal foundation models has unveiled a promising direction by leveraging well-understood data modalities. This tutorial focuses on existing representation learning works, from single-sensor approaches to cross-device and cross-modality pipelines. Furthermore, we will provide an overview of recent developments in multi-modal foundation models, which originated from language and vision learning, but have recently started incorporating inertial measurement units (IMU) and time-series data. This tutorial will offer an important forum for researchers in the mobile sensing community to discuss future research directions in representation learning for HAR, and in particular, to identify potential avenues to incorporate the latest advancements in multi-modal foundation models, aiming to finally solve the long-standing activity recognition problem.Keywords
This publication has 15 references indexed in Scilit:
- Assessing the State of Self-Supervised Human Activity Recognition Using WearablesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022
- COCOAProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022
- What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?2022
- Self-Supervised Representation Learning: Introduction, advances, and challengesIEEE Signal Processing Magazine, 2022
- ColloSSLProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022
- Contrastive Predictive Coding for Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2021
- IMUTubeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2020
- On the role of features in human activity recognition2019
- Multi-task Self-Supervised Learning for Human Activity DetectionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2019
- A tutorial on human activity recognition using body-worn inertial sensorsACM Computing Surveys, 2014