Sunwoo Lee
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Sunwoo Lee has a total of 65 co-authors. Here are the top co-authors by number of shared publications:
Salman Avestimehr
University of Southern California, Los Angeles, CA, USA
6 shared publicationsJihyun Lim
Department of Computer Engineering, Inha University, Incheon, Republic of Korea
3 shared publications- Salman Avestimehr
Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, USA
3 shared publications Alok Choudhary
2 shared publicationsAnkit Agrawal
2 shared publicationsChaoyang He
2 shared publications
Latest Publications
- Open Access
- 19 March 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 17 (2) , 1-27
Multi-Metric Client Activation Method for Fast and Accurate Federated Learning
While unbiased gradient estimators ensure unbiased solutions in empirical risk minimization problems, they can significantly hinder optimization efficiency and generalization performance in strongly non-IID Federated Learning environments. Although recent studies have demonstrated promising applications of biased estimators, they typically focus only on convergence rates while overlooking generalization performance. We propose a novel multi-metric bias concept, quantified using both local loss and local gradient norm, along with a client activation method based on this bias concept. The proposed method prioritizes training on local datasets that better represent the global dataset, leading to faster convergence and improved generalization. Our extensive empirical study demonstrates that carefully injecting bias into client activation accelerates federated optimization, achieving a substantially improved validation accuracy within a given epoch budget. In representative machine learning benchmarks, our method achieves up to \(12.7\%\) higher accuracy than uniform random sampling and \(2.5\%\) higher accuracy than state-of-the-art biased client activation methods.- 19 March 2026
- Published by Association for Computing Machinery (ACM) in ACM Transactions on Intelligent Systems and Technology
- Vol. 17 (2) , 1-27
Open AccessMulti-Metric Client Activation Method for Fast and Accurate Federated Learning
While unbiased gradient estimators ensure unbiased solutions in empirical risk minimization problems, they can significantly hinder optimization efficiency and generalization performance in strongly non-IID Federated Learning environments. Although recent studies have demonstrated promising applications of biased estimators, they typically focus only on convergence rates while overlooking generalization performance. We propose a novel multi-metric bias concept, quantified using both local loss and local gradient norm, along with a client activation method based on this bias concept. The proposed method prioritizes training on local datasets that better represent the global dataset, leading to faster convergence and improved generalization. Our extensive empirical study demonstrates that carefully injecting bias into client activation accelerates federated optimization, achieving a substantially improved validation accuracy within a given epoch budget. In representative machine learning benchmarks, our method achieves up to \(12.7\%\) higher accuracy than uniform random sampling and \(2.5\%\) higher accuracy than state-of-the-art biased client activation methods. - Open Access
Weight Variance Amplifier Improves Accuracy in High-Sparsity One-Shot Pruning
Deep neural networks achieve outstanding performance in visual recognition tasks, yet their large number of parameters makes them less practical for real-world applications. Recently, one-shot pruning has emerged as an effective strategy for reducing model size without additional training. However, models trained with standard objective functions often suffer a significant drop in accuracy after aggressive pruning. Some existing pruning-robust optimizers, such as SAM, and CrAM, mitigate this accuracy drop by guiding the model toward flatter regions of the parameter space, but they inevitably incur non-negligible additional computations. We propose a Variance Amplifying Regularizer (VAR) that deliberately increases the variance of model parameters during training. Our study reveals an intriguing finding that parameters with higher variance exhibit greater pruning robustness. VAR exploits this property by promoting such variance in the weight distribution, thereby mitigating the adverse effects of pruning. We further provide a theoretical analysis of its convergence behavior, supported by extensive empirical results demonstrating the superior pruning robustness of VAR.
Open AccessWeight Variance Amplifier Improves Accuracy in High-Sparsity One-Shot Pruning
Deep neural networks achieve outstanding performance in visual recognition tasks, yet their large number of parameters makes them less practical for real-world applications. Recently, one-shot pruning has emerged as an effective strategy for reducing model size without additional training. However, models trained with standard objective functions often suffer a significant drop in accuracy after aggressive pruning. Some existing pruning-robust optimizers, such as SAM, and CrAM, mitigate this accuracy drop by guiding the model toward flatter regions of the parameter space, but they inevitably incur non-negligible additional computations. We propose a Variance Amplifying Regularizer (VAR) that deliberately increases the variance of model parameters during training. Our study reveals an intriguing finding that parameters with higher variance exhibit greater pruning robustness. VAR exploits this property by promoting such variance in the weight distribution, thereby mitigating the adverse effects of pruning. We further provide a theoretical analysis of its convergence behavior, supported by extensive empirical results demonstrating the superior pruning robustness of VAR.
- Open Access
GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
Parameter-efficient fine-tuning (PEFT) has become a popular way to adapt large pre-trained models to new tasks. Most PEFT methods update only a small subset of parameters while freezing the rest, avoiding redundant computation. As they maximize the absolute size of the updates without regard to the parameters' original scale, the resulting changes in model behavior can be minimal. In contrast, we maximize updates relative to each parameter's scale, yielding more meaningful downstream adaptation. We propose Gradient-to-Weight Ratio and Entropy-guided Masking (GEM), a parameter scale-aware, distribution-sensitive sparse fine-tuning framework. GEM prioritizes parameters whose updates are significant in proportion to their initial pre-trained values. It also adaptively determines how many parameters to tune at each layer based on the entropy of parameter values, thereby making the most effective use of the computational budget in PEFT. Our empirical study demonstrates the efficacy of GEM on both general-domain tasks (GLUE and SuperGLUE) and domain-specific tasks (GSM8k and MBPP), achieving up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters.
Open AccessGEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
Parameter-efficient fine-tuning (PEFT) has become a popular way to adapt large pre-trained models to new tasks. Most PEFT methods update only a small subset of parameters while freezing the rest, avoiding redundant computation. As they maximize the absolute size of the updates without regard to the parameters' original scale, the resulting changes in model behavior can be minimal. In contrast, we maximize updates relative to each parameter's scale, yielding more meaningful downstream adaptation. We propose Gradient-to-Weight Ratio and Entropy-guided Masking (GEM), a parameter scale-aware, distribution-sensitive sparse fine-tuning framework. GEM prioritizes parameters whose updates are significant in proportion to their initial pre-trained values. It also adaptively determines how many parameters to tune at each layer based on the entropy of parameter values, thereby making the most effective use of the computational budget in PEFT. Our empirical study demonstrates the efficacy of GEM on both general-domain tasks (GLUE and SuperGLUE) and domain-specific tasks (GSM8k and MBPP), achieving up to a 1.6% improvement in fine-tuning accuracy over full fine-tuning while updating only 0.1% of model parameters.
- Open Access
- 1 July 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-10
Highly stable two-level current fluctuation in complex oxide heterostructures
Two-level systems based on point defects in dielectric oxides offer promising entropy source for random number generators. The random telegraph noise (RTN) generated by the two-level systems is ideal for creating random bit-strings for advanced computing and cryptographic technologies. However, in classical oxide systems, RTN signals often suffer from instability due to undesired defect migration and metastable electronic states. Herein, we present a two-level quantum system based on SrRuO3/LaAlO3/Nb-doped SrTiO3 heterostructure, which incorporates two different types of point defects, oxygen vacancies and antisite Ti defects. Temporal electron localization at antisite defects alters the energy levels of nearby oxygen vacancies through instantaneous Coulomb interaction, resulting in two-level current fluctuation across the interface. The RTN-like current signals exhibit high stability at room temperature. We utilize the stable two-level fluctuations to generate random bit-strings and confirm their applicability in practical stochastic machine learning algorithms for image super-resolution. This study provides a guideline for designing reliable entropy sources by exploiting the complementary interactions between cation and anion point defects in oxide-based electronic systems, essential for hardware-based random number generators.
- 1 July 2025
- Published by Springer Nature in Nature Communications
- Vol. 16 (1) , 1-10
Open AccessHighly stable two-level current fluctuation in complex oxide heterostructures
Two-level systems based on point defects in dielectric oxides offer promising entropy source for random number generators. The random telegraph noise (RTN) generated by the two-level systems is ideal for creating random bit-strings for advanced computing and cryptographic technologies. However, in classical oxide systems, RTN signals often suffer from instability due to undesired defect migration and metastable electronic states. Herein, we present a two-level quantum system based on SrRuO3/LaAlO3/Nb-doped SrTiO3 heterostructure, which incorporates two different types of point defects, oxygen vacancies and antisite Ti defects. Temporal electron localization at antisite defects alters the energy levels of nearby oxygen vacancies through instantaneous Coulomb interaction, resulting in two-level current fluctuation across the interface. The RTN-like current signals exhibit high stability at room temperature. We utilize the stable two-level fluctuations to generate random bit-strings and confirm their applicability in practical stochastic machine learning algorithms for image super-resolution. This study provides a guideline for designing reliable entropy sources by exploiting the complementary interactions between cation and anion point defects in oxide-based electronic systems, essential for hardware-based random number generators.
- Open Access
- 26 April 2025
- Published by Wiley in Advanced Science
- Vol. 12 (23) , e2417811
Machine Learning Approach to Characterize Ferromagnetic La 0.7 Sr 0.3 MnO 3 Thin Films via Featurization of Surface Morphology
Ferromagnetic perovskite oxides, particularly La 0.7 Sr 0.3 MnO 3 (LSMO), show significant promise for spintronics and electromagnetic applications due to their unique half‐metallicity and colossal magnetoresistance properties. These properties are known to arise from Mn‐O‐Mn double‐exchange interactions, which are directly related to microscopic lattice structures. However, since the microscopic structure in LSMO is highly sensitive to various material parameters, such as thickness, lattice strain, oxygen deficiency, and cation stoichiometry, understanding the intricate relationship between the microscopic structures and the resulting physical properties of LSMO remains challenging. Herein, a machine learning approach is introduced to characterize ferromagnetic LSMO thin films by featurization of their surface morphology. Using an ensemble machine learning method, the non‐linear correlations between surface morphology and the electronic, magnetic properties of LSMO thin films are captured and modeled. Based on these estimated correlations, LSMO thin films are classified into five representative types, each characterized by distinctive properties and surface morphologies. These results imply that surface morphology can reveal hidden information about the strongly correlated properties of ferromagnetic LSMO thin films. Consequently, the machine learning‐based approach provides an efficient method for understanding the correlated material properties of ferromagnetic oxides and related materials through surface morphology analysis.
- 26 April 2025
- Published by Wiley in Advanced Science
- Vol. 12 (23) , e2417811
Open AccessMachine Learning Approach to Characterize Ferromagnetic La 0.7 Sr 0.3 MnO 3 Thin Films via Featurization of Surface Morphology
Ferromagnetic perovskite oxides, particularly La 0.7 Sr 0.3 MnO 3 (LSMO), show significant promise for spintronics and electromagnetic applications due to their unique half‐metallicity and colossal magnetoresistance properties. These properties are known to arise from Mn‐O‐Mn double‐exchange interactions, which are directly related to microscopic lattice structures. However, since the microscopic structure in LSMO is highly sensitive to various material parameters, such as thickness, lattice strain, oxygen deficiency, and cation stoichiometry, understanding the intricate relationship between the microscopic structures and the resulting physical properties of LSMO remains challenging. Herein, a machine learning approach is introduced to characterize ferromagnetic LSMO thin films by featurization of their surface morphology. Using an ensemble machine learning method, the non‐linear correlations between surface morphology and the electronic, magnetic properties of LSMO thin films are captured and modeled. Based on these estimated correlations, LSMO thin films are classified into five representative types, each characterized by distinctive properties and surface morphologies. These results imply that surface morphology can reveal hidden information about the strongly correlated properties of ferromagnetic LSMO thin films. Consequently, the machine learning‐based approach provides an efficient method for understanding the correlated material properties of ferromagnetic oxides and related materials through surface morphology analysis.
- Open Access
Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
Sharpness-Aware Minimization (SAM) is an optimization method that improves generalization performance of machine learning models. Despite its superior generalization, SAM has not been actively used in real-world applications due to its expensive computational cost. In this work, we propose a novel asynchronous-parallel SAM which achieves nearly the same gradient norm penalizing effect like the original SAM while breaking the data dependency between the model perturbation and the model update. The proposed asynchronous SAM can even entirely hide the model perturbation time by adjusting the batch size for the model perturbation in a system-aware manner. Thus, the proposed method enables to fully utilize heterogeneous system resources such as CPUs and GPUs. Our extensive experiments well demonstrate the practical benefits of the proposed asynchronous approach. E.g., the asynchronous SAM achieves comparable Vision Transformer fine-tuning accuracy (CIFAR-100) as the original SAM while having almost the same training time as SGD.
Open AccessAsynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning
Sharpness-Aware Minimization (SAM) is an optimization method that improves generalization performance of machine learning models. Despite its superior generalization, SAM has not been actively used in real-world applications due to its expensive computational cost. In this work, we propose a novel asynchronous-parallel SAM which achieves nearly the same gradient norm penalizing effect like the original SAM while breaking the data dependency between the model perturbation and the model update. The proposed asynchronous SAM can even entirely hide the model perturbation time by adjusting the batch size for the model perturbation in a system-aware manner. Thus, the proposed method enables to fully utilize heterogeneous system resources such as CPUs and GPUs. Our extensive experiments well demonstrate the practical benefits of the proposed asynchronous approach. E.g., the asynchronous SAM achieves comparable Vision Transformer fine-tuning accuracy (CIFAR-100) as the original SAM while having almost the same training time as SGD.
- Open Access
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1518-1529
Layer-Wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning
Sharpness-aware minimization (SAM) is known to improve the generalization performance of neural networks. However, it is not widely used in real-world applications yet due to its expensive model perturbation cost. A few variants of SAM have been proposed to tackle such an issue, but they commonly do not alleviate the cost noticeably. In this paper, we propose a lightweight layer-wise gradient norm penalizing method that tackles the expensive computational cost of SAM while maintaining its superior generalization performance. Our study empirically proves that the gradient norm of the whole model can be effectively suppressed by penalizing the gradient norm of only a few critical layers. We also theoretically show that such a partial model perturbation does not harm the convergence rate of SAM, allowing them to be safely adapted in real-world applications. To demonstrate the efficacy of the proposed method, we perform extensive experiments comparing the proposed method to mini-batch SGD and the conventional SAM using representative computer vision and language modeling benchmarks.
- 24 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 1518-1529
Open AccessLayer-Wise Adaptive Gradient Norm Penalizing Method for Efficient and Accurate Deep Learning
Sharpness-aware minimization (SAM) is known to improve the generalization performance of neural networks. However, it is not widely used in real-world applications yet due to its expensive model perturbation cost. A few variants of SAM have been proposed to tackle such an issue, but they commonly do not alleviate the cost noticeably. In this paper, we propose a lightweight layer-wise gradient norm penalizing method that tackles the expensive computational cost of SAM while maintaining its superior generalization performance. Our study empirically proves that the gradient norm of the whole model can be effectively suppressed by penalizing the gradient norm of only a few critical layers. We also theoretically show that such a partial model perturbation does not harm the convergence rate of SAM, allowing them to be safely adapted in real-world applications. To demonstrate the efficacy of the proposed method, we perform extensive experiments comparing the proposed method to mini-batch SGD and the conventional SAM using representative computer vision and language modeling benchmarks.
- 22 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (12) , 11133-11143
Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training
In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. To implement large-scale FL applications, thus, it is crucial to develop a distributed learning method that enables the participation of such weak clients. We propose$\mathtt{EmbracingFL}$ , a general FL framework that allows all available clients to join the distributed training regardless of their system resource capacity. The framework is built upon a novel form of partial model training method in which each client trains as many consecutive output-side layers as its system resources allow. Our study demonstrates that$\mathtt{EmbracingFL}$ encourages each layer to have similar data representations across clients, improving FL efficiency. The proposed partial model training method guarantees convergence to a neighbor of stationary points for non-convex and smooth problems. We evaluate the efficacy of$\mathtt{EmbracingFL}$ under a variety of settings with a mixed number of strong, moderate ($\sim\! 40\%$ memory), and weak ($\sim\! 15\%$ memory) clients, datasets (CIFAR-10, FEMNIST, and IMDB), and models (ResNet20, CNN, and LSTM). Our empirical study shows that$\mathtt{EmbracingFL}$ consistently achieves high accuracy as like all clients are strong, outperforming the state-of-the-art width reduction methods (i.e., HeteroFL and FjORD).- 22 April 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (12) , 11133-11143
Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training
In Federated Learning (FL), clients may have weak devices that cannot train the full model or even hold it in their memory space. To implement large-scale FL applications, thus, it is crucial to develop a distributed learning method that enables the participation of such weak clients. We propose$\mathtt{EmbracingFL}$ , a general FL framework that allows all available clients to join the distributed training regardless of their system resource capacity. The framework is built upon a novel form of partial model training method in which each client trains as many consecutive output-side layers as its system resources allow. Our study demonstrates that$\mathtt{EmbracingFL}$ encourages each layer to have similar data representations across clients, improving FL efficiency. The proposed partial model training method guarantees convergence to a neighbor of stationary points for non-convex and smooth problems. We evaluate the efficacy of$\mathtt{EmbracingFL}$ under a variety of settings with a mixed number of strong, moderate ($\sim\! 40\%$ memory), and weak ($\sim\! 15\%$ memory) clients, datasets (CIFAR-10, FEMNIST, and IMDB), and models (ResNet20, CNN, and LSTM). Our empirical study shows that$\mathtt{EmbracingFL}$ consistently achieves high accuracy as like all clients are strong, outperforming the state-of-the-art width reduction methods (i.e., HeteroFL and FjORD).- Open Access
Federated Learning of Large Models at the Edge via Principal Sub-Model Training
Federated Learning (FL) is emerging as a popular, promising decentralized learning framework that enables collaborative training among clients, with no need to share private data between them or to a centralized server. However, considering many edge clients do not have sufficient computing, memory, or communication capabilities, federated learning of large models still faces significant bottlenecks. To keep such weak but crucial clients in the loop, prior works either consider a heterogeneous-client setting where clients train models with different sizes; or offload training to the server. However, the heterogeneous-client setting requires some clients to train full model, which is not aligned with the resource-constrained setting; while the latter ones break privacy promises in FL when sharing intermediate representations or labels with the server. To overcome these limitations, in this work, we formulate a realistic, but much less explored, cross-device FL setting in which no client can train a full large model nor is willing to share any intermediate information with the remote server. Under such a formulation, we develop a principal sub-model (PriSM) training methodology to collaboratively train a full large model, while assigning each client a small sub-model that is a probabilistic low-rank approximation to the full server model. When creating sub-models, PriSM first performs a principal kernel analysis in the orthogonal kernel space to obtain importance of each kernel. Then, PriSM adopts a novel importance-aware sampling process to select a subset of kernels (i.e., a kernel with high importance is assigned with a higher sampling probability). This sampling process ensures each sub-model is still a low-rank approximation to the full model, while all sub-models together achieve nearly full coverage on the principal kernels.
Open AccessFederated Learning of Large Models at the Edge via Principal Sub-Model Training
Federated Learning (FL) is emerging as a popular, promising decentralized learning framework that enables collaborative training among clients, with no need to share private data between them or to a centralized server. However, considering many edge clients do not have sufficient computing, memory, or communication capabilities, federated learning of large models still faces significant bottlenecks. To keep such weak but crucial clients in the loop, prior works either consider a heterogeneous-client setting where clients train models with different sizes; or offload training to the server. However, the heterogeneous-client setting requires some clients to train full model, which is not aligned with the resource-constrained setting; while the latter ones break privacy promises in FL when sharing intermediate representations or labels with the server. To overcome these limitations, in this work, we formulate a realistic, but much less explored, cross-device FL setting in which no client can train a full large model nor is willing to share any intermediate information with the remote server. Under such a formulation, we develop a principal sub-model (PriSM) training methodology to collaboratively train a full large model, while assigning each client a small sub-model that is a probabilistic low-rank approximation to the full server model. When creating sub-models, PriSM first performs a principal kernel analysis in the orthogonal kernel space to obtain importance of each kernel. Then, PriSM adopts a novel importance-aware sampling process to select a subset of kernels (i.e., a kernel with high importance is assigned with a higher sampling probability). This sampling process ensures each sub-model is still a low-rank approximation to the full model, while all sub-models together achieve nearly full coverage on the principal kernels.
- 8 August 2023
- Published by Elsevier in Neurocomputing
- Vol. 556
Partial model averaging in Federated Learning: Performance guarantees and benefits
- 8 August 2023
- Published by Elsevier in Neurocomputing
- Vol. 556
Partial model averaging in Federated Learning: Performance guarantees and benefits