A Robust Aggregation of Federated Large Language Models for Multimodal Knowledge Discovery in Computational Social Systems
- 8 July 2025
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Computational Social Systems
- Vol. 12 (6) , 5433-5448
Abstract
Amid a rapidly evolving information era, large-scale multimodal knowledge discovery in computational social systems emerges as a key research domain. Large language models (LLMs) play a crucial role in this field, providing contextual understanding and task adaptability. Yet, centralized training of LLM raises privacy concerns. Federated learning (FL) offers a distributed alternative, but it struggles with data heterogeneity and security issues related to model parameters. To this end, we propose a robust aggregation method that leverages the relative total distance of models to improve global model performance in heterogeneous settings, complemented by Cheon-Kim-Kim-Song (CKKS) encryption to secure parameters against parameter stealing without performance loss. Extensive numeric results show our approach excels in LLM testing, scoring 3.74 on MTBenchmark and 8.17 on Vicuna, outperforming state-of-the-art FL methods against data heterogeneity challenges. It also achieves consistent gains on image datasets such as SVHN, CIFAR10, MNIST, TinyImageNet200, and CIFAR100, TinyImageNet200. In summary, our method offers an effective solution for secure multimodal data analysis in computational social systems.Keywords
Funding Information
- JSPS KAKENHI (JP25K00373)
- Ongoing Research Funding Program (ORF-2025-102)
This publication has 31 references indexed in Scilit:
- LLM Enhanced Representation for Cold Start Service Recommendation2024
- Influence of External Information on Large Language Models Mirrors Social Cognitive PatternsIEEE Transactions on Computational Social Systems, 2024
- A survey on large language model (LLM) security and privacy: The Good, The Bad, and The UglyHigh-Confidence Computing, 2024
- A Novel Fake News Detection Model for Context of Mixed Languages Through Multiscale TransformerIEEE Transactions on Computational Social Systems, 2023
- Confidence-Aware Personalized Federated Learning via Variational Expectation Maximization2023
- FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning2023
- A Privacy-Preserving Social Computing Framework for Health Management Using Federated LearningIEEE Transactions on Computational Social Systems, 2022
- Multi-center federated learning: clients clustering for better personalizationWorld Wide Web, 2022
- Are We There Yet? Timing and Floating-Point Attacks on Differential Privacy Systems2022
- Homomorphic Encryption for Arithmetic of Approximate Numbers2017