The Case for Micro Foundation Models to Support Robust Edge Intelligence
Authors:
- 28 October 2024
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 23-31
Abstract
This paper advocates the concept of micro foundation models (μFMs), recently introduced by the authors to describe a category of self-supervised pre-training solutions that we argue are necessary to support robust intelligent inference tasks in Internet of Things (IoT) applications. The work is motivated by the fact that collecting sufficient amounts of labeled data in IoT applications to train AI/ML tasks is challenging due to the difficulties in labeling such data after the fact. In the absence of sufficient labeled data, supervised training solutions become brittle and prone to overfitting. Self-supervised training obviates the collection of labeled data, allowing pre-training with the more readily available unlabeled data instead. Specifically, the μFMs discussed in this paper use self-supervised pre-training to develop an encoder that maps input data into a semantically-organized latent representation in a manner agnostic to the downstream inference task. Our preliminary work shows that this (unsupervised) encoder can be moderately sized, yet produce a latent representation that simultaneously supports the fine-tuning of multiple downstream inference tasks, each at a minimal labeling cost. We demonstrate the efficacy of this pre-training/fine-tuning pipeline using a vibration-based μFM as a running case study. The study shows that the fine-tuning of inference tasks on top of the aforementioned encoder-produced latent representation needs orders of magnitude fewer labels than supervised training solutions, and that the resulting tasks are significantly more robust to environmental changes and easier to adapt to domain shifts compared to their supervised counterparts. Furthermore, we show that inference algorithms based on our example μFM can be executed in real time on a Raspberry Pi device, making the approach viable for the IoT space. We conclude that μFMs are a preferred (and likely necessary) route to support robust intelligent sensing on IoT devices in subareas where labeled data collection is challenging. The paper is a call for the research community to invest in μFM research for IoT applications.Keywords
This publication has 30 references indexed in Scilit:
- Acies-OS: A Content-Centric Platform for Edge AI Twinning and Orchestration2024
- Understanding Masked Autoencoders via Hierarchical Latent Variable Models2023
- Robust Automatic Modulation Classification in Low Signal to Noise RatioIEEE Access, 2023
- Masked Autoencoders Are Scalable Vision Learners2022
- SimMIM: a Simple Framework for Masked Image Modeling2022
- Swin Transformer: Hierarchical Vision Transformer using Shifted Windows2021
- An Empirical Study of Training Self-Supervised Vision Transformers2021
- Contrastive Multimodal Fusion with TupleInfoNCE2021
- Contrastive Multiview Coding2020
- Degeneracy and networked buffering: principles for supporting emergent evolvability in agile manufacturing systemsNatural Computing, 2011