THOR: Thermal-Guided Hand-Object Reasoning via Adaptive Vision Sampling
Open Access
- 2 December 2025
- journal article
- Published by Association for Computing Machinery (ACM) in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
- Vol. 9 (4) , 1-25
Abstract
Wearable cameras are increasingly used as an observational and interventional tool for human behaviors by providing detailed visual data of hand-related activities. This data can be leveraged to facilitate memory recall for logging of behavior or timely interventions aimed at improving health. However, continuous processing of RGB images from these cameras consumes significant power impacting battery lifetime, generates a large volume of unnecessary video data for post-processing, raises privacy concerns, and requires substantial computational resources for real-time analysis. We introduce THOR, a real-time adaptive spatio-temporal RGB frame sampling method that leverages thermal sensing to capture hand-object patches and classify them in real time. We use low-resolution thermal camera data to identify moments when a person switches from one hand-related activity to another and adjust the RGB frame sampling rate by increasing it during activity transitions and reducing it during periods of sustained activity (when the system has enough information to identify the activity). Additionally, we use the thermal cues from the hand to localize the region of interest (i.e. , the hand-object interaction) in each RGB frame, allowing the system to crop and process only the necessary part of the image for activity recognition. We develop a wearable device to validate our method through an in-the-wild study with 14 participants and over 30 activities, and further evaluate it on Ego4D (923 participants across 9 countries, totaling 3,670 hours of video). Our results show that using only 3% of the original RGB video data, our method captures all the activity segments, and achieves a hand-related activity recognition F1-score (95%) comparable to using the entire RGB video (94%). Our work provides a more practical path for the longitudinal use of wearable cameras to monitor hand-related activities and health-risk behaviors in real time.Keywords
All Related Versions
Funding Information
- National Institute of Diabetes and Digestive and Kidney Diseases (R01DK129843)
- National Institute of Biomedical Imaging and Bioengineering (R21EB030305)
This publication has 21 references indexed in Scilit:
- FastPicker: Adaptive independent two-stage video-to-video summarization for efficient action recognitionNeurocomputing, 2022
- Ego4D: Around the World in 3,000 Hours of Egocentric Video2022
- FrameExit: Conditional Early Exiting for Efficient Video Recognition2021
- Overview of Lifelogging: Current Challenges and AdvancesIEEE Access, 2021
- Selective Content Removal for Egocentric Wearable Camera in Nutritional StudiesIEEE Access, 2020
- Understanding Human Hands in Contact at Internet Scale2020
- AdaFrame: Adaptive Frame Selection for Fast Video Recognition2019
- MobileNetV2: Inverted Residuals and Linear Bottlenecks2018
- Image-Assisted Dietary Assessment: A Systematic Review of the EvidenceJournal of the Academy of Nutrition and Dietetics, 2015
- ImageNet: A large-scale hierarchical image database2009