Jian Lou

Fine-tuning large language models (LLMs) on down-stream tasks has become a standard approach to adapt their capabilities. However, the process raises privacy concerns when using sensitive datasets, prompting increasing interest in differentially private (DP) fine-tuning methods. While existing approaches build upon the seminal work of DP-SGD, they are constrained by the inherent inefficiency bottlenecks. In this paper, we investigate the potential of DP zeroth-order methods for LLM fine-tuning, which avoids the scalability bottleneck of SGD by approximating gradients with more efficient zeroth-order gradients. We propose the stagewise DP zeroth-order method (DP-ZOSO) that dynamically schedules key hyperparameters to leverage the synergy between DP random perturbation and the gradient approximation error. To further enhance the scalability, we propose DP zeroth-order stagewise pruning method (DP-ZOPO) which reduces the trainable parameters by a data-free pruning technique requiring no extra privacy budget. We provide theoretical analysis for both proposed methods and conduct extensive empirical analysis on both encoder-only masked and decoder-only autoregressive language model, achieving impressive results in terms of scalability and utility across diverse tasks.

Federated Semi-Supervised Learning (FSSL) aims to learn a global model from different clients in an environment with both labeled and unlabeled data. Most of the existing FSSL work generally assumes that both types of data are available on each client. In this paper, we study a more general problem setup of FSSL with annotation heterogeneity, where each client can hold an arbitrary percentage (0%-100%) of labeled data. To this end, we propose a novel FSSL framework called Heterogeneously Annotated Semi-Supervised LEarning (HASSLE). Specifically, it employs a dual-model approach. Two models with the same architecture are trained separately: one uses labeled data only, and the other uses unlabeled data. This design enables the framework to be applied to clients with arbitrary labeling percentages. Furthermore, a mutual learning strategy called Supervised-Unsupervised Mutual Alignment (SUMA) is proposed for the dual models with global residual alignment and model proximity alignment. Subsequently, the dual models can implicitly learn from both types of data across different clients, although each dual model is only trained locally on a single type of data. Experiments verify that the dual models in HASSLE learned by SUMA can mutually learn from each other, thereby effectively utilizing the information of both types of data across different clients. Our code is available at https://github.com/shangxinyi/HASSLE.

Principal Component Analysis (PCA) remains a fundamental technique for unsupervised dimensionality reduction. However, its traditional centralized implementation poses challenges in the context of increasing data privacy concerns, particularly in scenarios involving vertically partitioned data across multiple clients. To address these challenges, we introduce VFedPCA and VFedAKPCA, pioneering federated algorithms designed for linear and nonlinear PCA in such distributed environments. These algorithms facilitate collaborative PCA computations across distributed clients while preserving data privacy by avoiding raw data exchanges. VFed- PCA leverages a local power iteration strategy enhanced by a warm-start mechanism to accelerate convergence. Meanwhile, VFedAKPCA innovatively extends this approach to kernel spaces, employing a novel weight-scaling technique for effective nonlinear feature extraction. We validate the efficacy of our proposed methods through extensive experiments on five real-world datasets, evaluating both server-client and peer-to-peer architectures. Our results indicate that these federated approaches achieve performance on par with traditional centralized PCA methods. The implementation code is publicly accessible at https://github.com/juyongjiang/VFedAKPCA">https://github.com/juyongjiang/VFedAKPCA, facilitating further research and application.