Training on the Fly: On-Device Self-Supervised Learning Aboard Nano-Drones Within 20 mW
- 27 August 2024
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems
- Vol. 43 (11) , 3685-3695
Abstract
Miniaturized cyber-physical systems (CPSes) powered by tiny machine learning (TinyML), such as nano-drones, are becoming an increasingly attractive technology. Their small form factor (i.e., 10diameter) ensures vast applicability, ranging from the exploration of narrow disaster scenarios to safe human-robot interaction. Simple electronics make these CPSes inexpensive, but strongly limit the computational, memory, and sensing resources available on board. In real-world applications, these limitations are further exacerbated by domain shift. This fundamental machine learning problem implies that model perception performance drops when moving from the training domain to a different deployment one. To cope with and mitigate this general problem, we present a novel on-device fine-tuning approach that relies only on the limited ultra-low power resources available aboard nano-drones. Then, to overcome the lack of ground-truth training labels aboard our CPS, we also employ a self-supervised method based on ego-motion consistency. Albeit our work builds on top of a specific real-world vision-based human pose estimation task, it is widely applicable for many embedded TinyML use cases. Our 512-image on-device training procedure is fully deployed aboard an ultra-low power GWT GAP9 System-on-Chip and requires only 1of memory while consuming as low as 19or running in just 510(at 38). Finally, we demonstrate the benefits of our on-device learning approach by field-testing our closed-loop CPS, showing a reduction in horizontal position error of up to 26% vs. a non-fine-tuned state-of-the-art baseline. In the most challenging never-seen-before environment, our on-device learning procedure makes the difference between succeeding or failing the mission.Keywords
All Related Versions
Funding Information
- Hasler Foundation (23059)
- Swiss National Science Foundation (213074)
This publication has 25 references indexed in Scilit:
- XVO: Generalized Visual Odometry via Cross-Modal Self-Training2023
- Reduced precision floating-point optimization for Deep Neural Network On-Device Learning on microcontrollersFuture Generation Computer Systems, 2023
- NanoFlowNet: Real-time Dense Optical Flow on a Nano Quadcopter2023
- Tiny-PULP-Dronets: Squeezing Neural Networks for Faster and Lighter Inference on Multi-Tasking Autonomous Nano-Drones2022
- TinyOL: TinyML with Online-Learning on Microcontrollers2021
- D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry2020
- Digging Into Self-Supervised Monocular Depth Estimation2019
- SelFlow: Self-Supervised Learning of Optical Flow2019
- Geometric Consistency for Self-Supervised End-to-End Visual Odometry2018
- Learning to fly by crashing2017