Ubiquitous memory augmentation via mobile multimodal embedding system
Open Access
- 19 June 2025
- journal article
- research article
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-12
Abstract
Forgetting is inevitable in human memory. Recently, multimodal embedding models have been proposed to vectorize multimodal reality into a unified embedding space. Once generated, these embeddings allow mobile users to quickly retrieve relevant information, effectively augmenting their memory. However, as the model’s capacity increases, its resource consumption also rises. The resulting slow throughput and significant computational resource requirements hinder its deployment on mobile devices. In this paper, we present Reminisce, an efficient on-device multimodal embedding system that enables high-throughput embedding and precise retrieval on resource-constrained mobile devices. The core design draws inspiration from the memory functions of the human brain, utilizing coarse-grained embeddings to identify likely candidates, which are then refined through query-driven fine-grained retrieval. A series of algorithm-hardware orchestrated optimizations automatically navigates this process and strengthen the embedding quality. Experiments show that Reminisce provides high-quality embedding representation with high throughput while operating silently in the background with negligible memory usage and reduced energy consumption.Keywords
All Related Versions
This publication has 28 references indexed in Scilit:
- ImageBind One Embedding Space to Bind Them All2023
- Towards artificial general intelligence via a multimodal foundation modelNature Communications, 2022
- Speformer: An Efficient Hardware-Software Cooperative Solution for Sparse Spectral Transformer2022
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning2021
- Introduction to NVIDIA Jetson Nano2020
- Clotho: an Audio Captioning Dataset2020
- Human Activity Recognition Using Smartphones2018
- BranchyNet: Fast inference via early exiting from deep neural networks2016
- Microsoft COCO: Common Objects in Context2014
- Top-down predictions in the cognitive brainBrain and Cognition, 2007