Enhancing Recommendation Capabilities Using Multi-Head Attention-Based Federated Knowledge Distillation
Open Access
- 1 May 2023
- journal article
- research article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 11, 45850-45861
Abstract
As the internet and mobile computing have advanced, recommendation algorithms are used to manage large amounts of data. However, traditional recommendation systems usually require collecting user data on a central server, which may expose user privacy. Furthermore, data and models from different organizations may be proprietary and cannot be shared directly, leading to data isolation. To address these challenges, we propose a method that combines federated learning (FL) with the recommendation system using a federated knowledge distillation algorithm based on a multi-head attention mechanism. In the proposed approach, knowledge distillation is introduced on the basis of FL to induce the training of the local network and facilitate knowledge transfer. Further, to address the non-independent identical distribution of training samples in FL, Wasserstein distance and regularization terms are incorporated into the objective function of federated knowledge distillation to reduce the distribution difference between server and client networks. A multi-head attention mechanism is used to enhance user encoding information. A combined adaptive learning rate is adopted to further improve the convergence. Compared to the benchmark model, this approach demonstrates significant improvements, with accuracy enhanced up to 10%, model training time shortened by approximately 45%, and average error and NDCG values decreased by 10%.Keywords
Funding Information
- Ministry of Science and ICT (2021R1A5A1021944)
- Basic Science Research Program funded by the Ministry of Education (NRF-2021R1I1A3043889)
This publication has 37 references indexed in Scilit:
- Deep Interest Evolution Network for Click-Through Rate PredictionProceedings of the AAAI Conference on Artificial Intelligence, 2019
- Federated Reinforcement Learning for Fast Personalization2019
- Privacy Enhanced Matrix Factorization for Recommendation with Local Differential PrivacyIEEE Transactions on Knowledge and Data Engineering, 2018
- RETRACTED ARTICLE: Hybrid Recommendation System for Heart Disease Diagnosis based on Multiple Kernel Learning with Adaptive Neuro-Fuzzy Inference SystemMultimedia Tools and Applications, 2017
- How the GDPR Will Change the WorldEuropean Data Protection Law Review, 2016
- Sequence-Level Knowledge Distillation2016
- The MovieLens DatasetsACM Transactions on Interactive Intelligent Systems, 2015
- Significance and Challenges of Big Data ResearchBig Data Research, 2015
- Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensionsIEEE Transactions on Knowledge and Data Engineering, 2005
- Content-based book recommending using learning for text categorization2000