DA-Distill: Dual-Alignment Distillation for Multimodal Knowledge Transfer on Edge Devices
- 13 August 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Consumer Electronics
- Vol. 71 (3) , 8055-8066
Abstract
The growing demand for multimodal functionalities in consumer electronics has driven the widespread adoption of multimodal pre-trained models, such as CLIP, enhancing human-computer interactions on edge devices through adaptive and context-aware functionalities. However, deep neural architectures impose significant computational complexity, storage demands, and data privacy challenges that limit traditional finetuning methods at the edge. To overcome these constraints, we propose an efficient knowledge transfer framework comprising two main modules: a data filtering module employing a combined confidence-driven and random sampling strategy to select representative samples while ensuring privacy compliance, and a knowledge distillation module that explicitly transfers multimodal representations via contrastive learning. Specifically, we construct bidirectional cross-modal contrastive distributions and minimize their Kullback-Leibler divergence to achieve representation alignment between the old and new models. Experimental results indicate that our method significantly outperforms conventional fine-tuning, achieving more than 1.5 accuracy improvement using only 5% of the original training data. Additionally, we identify optimal data selection strategies tailored to different data availability scenarios, demonstrating the practical effectiveness and flexibility of our approach for resource-constrained edge computing applications.Keywords
Funding Information
- National Natural Science Foundation of China (62206015, 62227801)
- Fundamental Research Funds for the Central Universities (FRF-KST-25-008)
- Young Scientist Program of The National New Energy Vehicle Technology Innovation Center
This publication has 28 references indexed in Scilit:
- A cross Transformer for image denoisingInformation Fusion, 2023
- Combined scaling for zero-shot transfer learningNeurocomputing, 2023
- Foundation models for generalist medical artificial intelligenceNature, 2023
- Knowledge Distillation with the Reused Teacher Classifier2022
- Robust fine-tuning of zero-shot models2022
- Natural Adversarial Examples2021
- Distilling Object Detectors With Fine-Grained Feature Imitation2019
- Relational Knowledge Distillation2019
- Structured Knowledge Distillation for Semantic Segmentation2019
- ImageNet Large Scale Visual Recognition ChallengeInternational Journal of Computer Vision, 2015