CMA: Cross-Modal Association Between Wearable and Structural Vibration Signal Segments for Indoor Occupant Sensing
Open Access
- 9 May 2023
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 96-109
Abstract
Indoor occupant sensing enables many smart home applications, and various sensing systems have been explored. Based on their installation requirements, we consider two categories of sensors – on- and off-body – and we look into the combination of them for occupant sensing due to their spatial and temporal complementarity. We focus on an example modality pair of wearable IMU and structural vibration that demonstrate modality complementarity in prior work. However, current efforts are built upon the assumption that the knowledge of the signal segments from two modalities are known, which is challenged in a multiple occupants co-living scenario. Therefore, establishing accurate cross-modal signal segment associations is essential to ensure that a correct complementary relationship is learned. We present CMA, a cross-modal signal segment association scheme between structural vibration and wearable sensors. We propose AD-TCN, a framework built upon a temporal convolutional network that calculates the amount of shared context between an structural vibration sensor and associated wearable sensor candidates from the parameters of the trained model. We evaluate CMA via a public multimodal dataset for systematic evaluation, and we collect a continuous uncontrolled dataset for robustness evaluation. CMA achieves up to AUC value, F1 score, and accuracy improvement compared to baselines.Keywords
Funding Information
- CITRIS and the Banatao Institute at the University of California
- American Psychological Foundation (APF) 2021 Drs. Rosalee G. and Raymond A. Weiss
This publication has 25 references indexed in Scilit:
- IDroneProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2018
- UniverSense2018
- FootprintIDProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2017
- Xception: Deep Learning with Depthwise Separable Convolutions2017
- Temporal Convolutional Networks for Action Segmentation and Detection2017
- What is an ROC curve?Emergency Medicine Journal, 2017
- Deep Residual Learning for Image Recognition2016
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification2015
- On Early Stopping in Gradient Descent LearningConstructive Approximation, 2007
- Using AUC and accuracy in evaluating learning algorithmsIEEE Transactions on Knowledge and Data Engineering, 2005