Yang Xu

Recently, Split Federated Learning (SFL) advances typical Federated Learning (FL) and offers a feasible solution by splitting a full model into a bottom model (for clients) and a top model (for the server), which alleviates the computation and/or communication burden on clients. However, existing SFL works often assume sufficient labeled data on clients, which is usually impractical. Besides, non-IID data poses another challenge to ensure efficient model training. To our best knowledge, the above two data-related issues have not been simultaneously addressed in SFL. Herein, we review the distinct properties of SFL and develop a novel Semi-supervised SFL system, called SemiSFL. In SemiSFL, a student bottom model and a teacher bottom model are assigned to each client, while the server maintains the top parts and full versions of both the global and teacher models. Cross-entity (i.e., clients and the server) semi-supervised training and server-side supervised training are alternately conducted. During semi-supervised training, we explore regularizing the student features (produced by student bottom models) from different clients with the teacher features (produced by teacher bottom models) before conducting semi-supervised training, so as to improve the generalization ability of student bottom models. Moreover, our theoretical and experimental investigations reveal that the inconsistent training processes on labeled and unlabeled data have an influence on the effectiveness of SemiSFL. To mitigate training inconsistency, we develop an algorithm for dynamically regulating the frequency of performing server-side supervised training. Extensive experiments on benchmark models and datasets show that SemiSFL provides a 3.8× speed-up in training time, reduces the communication cost by about 70.3% while reaching the target accuracy, and achieves up to 5.8% improvement in accuracy under different non-IID settings compared to the state-of-the-art baselines.

Considering the hardware-friendly characteristics, structured pruning has emerged as an effective solution to reduce the resource requirements of large language models (LLMs) on resource-constrained devices. Since pruning a certain number of parameters will often reduce the model accuracy, fine-tuning is usually required to recover performance loss. However, fine-tuning relies on high device power and substantial data, making it unsuitable for on-device applications. Recent approaches propose post-training structured pruning techniques that do not require fine-tuning, following three granularities, i.e., neurons, attention heads, and layers. Unfortunately, the previous solutions still suffer from the high memory overhead of substructures’ importance evaluation at different granularities and the problem of only applying to models with specific structures, which limit their scope of applications. This paper proposes COMP, a lightweight and general post-training structured hybrid-granularity pruning method designed for on-device applications. COMP begins by assessing the importance of model layers and neurons individually using distributional distance and the matrix condition number. Subsequently, COMP performs fine-grained neuron pruning and simultaneously determines whether to apply coarse-grained layer pruning or not by comparing the outcomes of the two granularity strategies. Furthermore, COMP implements mask tuning to restore model accuracy without additional fine-tuning, minimizing dependency on external data resources and device power. Experimental results demonstrate that COMP performs well across models with various architectures. When pruning 20% of LLaMA-2-13B parameters, COMP requires less than 8 GB of memory, maintaining more than 90% model accuracy. Meanwhile, COMP improves performance by 3.54% compared to LLM-Pruner, while reducing memory overhead by 90%.

Recently, aided by edge computing (EC), federated meta-learning (FML) has been proposed to adequately utilize personalized data on different edge devices by incorporating meta-learning with federated learning (FL). Unlike conventional FL which conducts several stochastic gradient descent (SGD) steps to directly update the local model in each training round, FML first performs some SGD steps to obtain a reference model and then performs a meta-step to update the local model along the direction towards the reference model. In this way, FML can identify common patterns and structures across different clients (i.e., edge devices) and quickly adapt to new data distributions, which exhibits its advantage in dealing with statistical heterogeneity inherent in FL. However, existing FML approaches struggle to effectively cope with two key challenges: i) the disparity in dataset informativeness among clients with statistically heterogeneous local data; and ii) the disparity in computing capacities among clients with heterogeneous hardware configurations. Herein, we propose a heterogeneity-aware FML framework, termed HFML, which explores adjusting the execution frequencies of SGD step and meta-step in each training round to address the above statistical and system heterogeneity challenges simultaneously. We introduce usable information (UI) as a new metric in HFML to quantify dataset informativeness, and theoretically analyze the influence of the execution frequencies of SGD steps and meta-steps on model performance and convergence speed. Based on the theoretical analysis, we develop an efficient optimization algorithm to jointly adjust the execution frequencies for different clients to handle the heterogeneity issues, which contributes to accelerating model training and enhancing personalization performance. Experimental results demonstrate that HFML can speed up training by up to 4.02× and improve model accuracy by an average of 11.5% compared to the baselines under heterogeneous settings.

Mobile devices contribute more than half of the world’s web traffic, providing massive and diverse data for powering various computer vision (CV) applications, such as fall detection and image segmentation. To extract knowledge from remote devices without pulling their raw data, federated learning (FL) has emerged and usually works in a client-server fashion. To relax the computation burdens of training large-scale models on resource-constrained workers, we propose a novel split federated learning (SFL) framework with worker clustering and feature compression, termed ClusterSFL. All participating workers are supposed to train only the bottom or top submodel, and are arranged into multiple clusters. In each cluster, the workers collaboratively perform the SFL training procedure in parallel, while the PS only aggregates the cluster-wise entire models rather than massive features. However, considering the critical challenges encountered in on-device intelligent CV applications, such as communication resource limitation, statistical heterogeneity, and system heterogeneity, it is challenging to arrange suitable workers to specific clusters for efficient model training. Herein, we develop an effective worker clustering strategy and feature compression ratio configuration by optimizing the utility functions related to training efficiency and model accuracy. Specifically, ClusterSFL partitions workers with appropriate feature compression ratios into different clusters under the heterogeneity restrictions, thereby promoting model accuracy and training efficiency. Meanwhile, ClusterSFL adaptively assigns diverse local updating frequencies for each cluster to further address system heterogeneity. Extensive experiments are conducted on a physical platform with 80 NVIDIA Jetson devices, and the experimental results show that ClusterSFL reduces traffic consumption by about 22%∼63%, speeds up model training by about 1.29×∼2.83×, and improves model accuracy by about 2%∼27% in heterogeneous scenarios, compared to the baselines.

Community detection, an application of graph clustering in attributed networks, aims to identify closely related subsets within the network. Recently, graph neural networks (GNN) have advanced, enhancing feature extraction from graph data, leading to popular GNN-based community detection models. However, existing approaches often fall short in fully leveraging the topology and attribute information of the graph to learn rich node feature representations. To address this issue, we propose a model named Community Detection Based on Enhanced Deep Features in Attributed Networks (CDBAN). Firstly, we construct a Residual Auto-Encoder to learn hierarchical attribute information of nodes, addressing the gradient vanishing in linear fully connected layers and enriching attribute information. Subsequently, we develop a Residual Graph Auto-Encoder that integrates topology information and hierarchical attribute information from the Residual Auto-Encoder. Deep networks with residual connections capture better node features than deep networks without them, addressing the over-smoothing of node features caused by the deepening hierarchy of GNN. Furthermore, in order to enhance the Residual Graph Auto-Encoder's learning of hierarchical attribute information, we propose an Attribute Graph Attention Decoder to reconstruct attribute information from node features. Experimental results on four datasets show that our proposed CDBAN model has significant advantages.

Federated fine-tuning (FedFT) has been proposed to fine-tune the pre-trained language models in a distributed manner. However, there are two critical challenges for efficient FedFT in practical applications, i.e., resource constraints and system heterogeneity. Existing works rely on parameter-efficient fine-tuning methods, e.g., low-rank adaptation (LoRA), but with major limitations. Herein, based on the inherent characteristics of FedFT, we observe that LoRA layers with higher ranks added close to the output help to save resource consumption while achieving comparable fine-tuning performance. Then we propose a novel LoRA-based FedFT framework, termed LEGEND, which faces the difficulty of determining the number of LoRA layers (called, LoRA depth) and the rank of each LoRA layer (called, rank distribution). We analyze the coupled relationship between LoRA depth and rank distribution, and design an efficient LoRA configuration algorithm for heterogeneous devices, thereby promoting fine-tuning efficiency. Extensive experiments are conducted on a physical platform with 80 commercial devices. The results show that LEGEND can achieve a speedup of 1.5-2.8× and save communication costs by about 42.3% when achieving the target accuracy, compared to the advanced solutions.