Abstract
Multi-modal Federated Learning (FL) is a type of FL that considers utilizing multiple modalities of data to improve overall performance. While multi-modal data brings richer information, it also introduces more significant communication overhead. Reducing this overhead hinges on two key strategies: increasing the convergence speed of the training or reducing the communication overhead in each communication round. However, few studies have considered these two strategies simultaneously and formed a unified optimization framework. Thus, we propose a joint client and modality selection framework to reduce communication overhead. Modality selection executed on each client assigns weights to modalities based on their contribution to training potential, aiming at accelerating the convergence. Client selection executed on the server assigns weights to clients by considering different metrics, especially total training potential after the modality selection. We validate our proposed method on the five widely used open-source datasets, achieving satisfactory accuracy while reducing the total communication overhead to 2.43%–14.24% compared to without selection on different datasets, significantly outperforming existing state-of-the-art (SOTA) methods. Code is available at https://github.com/1643204431/OCETPMMFL .
Funding Information
  • National Natural Science Foundation of China (62202055 and 62202016)
  • Guangdong Basic and Applied Basic Research Foundation (2025A1515012843)
  • Start-up Fund from Beijing Normal University (312200502510)
  • Internal Fund from Beijing Normal-Hong Kong Baptist University (UICR0400003-24)
  • Project of Young Innovative Talents of Guangdong Education Department (2022KQNCX102)
  • Interdisciplinary Intelligence SuperComputer Center of Beijing Normal University

This publication has 18 references indexed in Scilit: