Lifeng Sun
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Lifeng Sun has a total of 114 co-authors. Here are the top co-authors by number of shared publications:
- Haitian Pang
Department of Computer Science and Technology, Tsinghua University, Beijing, China
12 shared publications - Shiqiang Yang
Department of Mechanical Engineering Xi'an University of Technology Xi'an Shaanxi China
9 shared publications - Tianchi Huang
Tsinghua University, Beijing, China
8 shared publications Tianchi Huang
Department of Computer Science and Technology, Tsinghua University, Beijing, China
7 shared publications- Rui-Xiao Zhang
Department of Computer Science, University of Illinois Urbana-Champaign, Urbana Champaign, IL, USA
6 shared publications - Jiangchuan Liu
Simon Fraser University, Burnaby, Canada
5 shared publications
Latest Publications
- Open Access
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 569-579
FedICL-KVc: Federated In-Context Learning via KV Cache
- 6 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 569-579
Open AccessFedICL-KVc: Federated In-Context Learning via KV Cache
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 25 (7) , 9997-10014
Toward Robust Low-Latency Live Streaming: Measurement, Prediction, and Rate Adaptation Under Uncertainty
- 1 July 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 25 (7) , 9997-10014
Toward Robust Low-Latency Live Streaming: Measurement, Prediction, and Rate Adaptation Under Uncertainty
- 25 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. PP (15361233) , 1-17
Federated LoRA Fine-Tuning of LLMs with Only Transmitting Matrix A or B
Recent advancements in large language models (LLMs) have been largely driven by vast amounts of training data. As public data are exhausted, fine-tuning LLMs with private data has become increasingly important. However, due to privacy concerns associated with sensitive data, traditional centralized training approaches are not feasible. Federated learning (FL) offers a promising solution, as it enables collaborative model training across multiple mobile devices without data exchange. Nevertheless, it is challenging to fine-tune LLMs with FL on resource-constrained and heterogeneous clients. To reduce the cost, we employ Low-Rank Adaptation (LoRA) for efficient fine-tuning. Existing federated LoRA fine-tuning methods suffer from inaccurate aggregation, and the resulting noise may lead to performance loss that may be more severe for LLMs. In this paper, we propose a novel method, FedOTAB, to mitigate this issue. FedOTAB first correctly aggregates all local LoRA modules and then fixes the matrix $\bf {A}$ or $\bf {B}$ to optimize the corresponding best approximate matrix $\bf {B}^*$ or $\bf {A}^*$ for each local LoRA module by minimizing the induced noise. In addition, we design a mechanism to freeze the matrices $\bf {A}$ and $\bf {B}$ alternately to stabilize optimization. By transmitting only the updated part, that is, half of the LoRA module, FedOTAB enjoys favorable communication efficiency. By explicitly splitting LoRA into local and global parts, FedOTAB effectively integrates both local and global knowledge. Extensive experiments demonstrate that FedOTAB excels in both homogeneous and heterogeneous rank scenarios, consistently outperforms state-of-the-art methods with remarkably low communication and computational overhead. We hope that this work provides valuable insights for federated LoRA fine-tuning, offering a practical and scalable solution to fine-tuning large models in mobile environments.
- 25 May 2026
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. PP (15361233) , 1-17
Federated LoRA Fine-Tuning of LLMs with Only Transmitting Matrix A or B
Recent advancements in large language models (LLMs) have been largely driven by vast amounts of training data. As public data are exhausted, fine-tuning LLMs with private data has become increasingly important. However, due to privacy concerns associated with sensitive data, traditional centralized training approaches are not feasible. Federated learning (FL) offers a promising solution, as it enables collaborative model training across multiple mobile devices without data exchange. Nevertheless, it is challenging to fine-tune LLMs with FL on resource-constrained and heterogeneous clients. To reduce the cost, we employ Low-Rank Adaptation (LoRA) for efficient fine-tuning. Existing federated LoRA fine-tuning methods suffer from inaccurate aggregation, and the resulting noise may lead to performance loss that may be more severe for LLMs. In this paper, we propose a novel method, FedOTAB, to mitigate this issue. FedOTAB first correctly aggregates all local LoRA modules and then fixes the matrix $\bf {A}$ or $\bf {B}$ to optimize the corresponding best approximate matrix $\bf {B}^*$ or $\bf {A}^*$ for each local LoRA module by minimizing the induced noise. In addition, we design a mechanism to freeze the matrices $\bf {A}$ and $\bf {B}$ alternately to stabilize optimization. By transmitting only the updated part, that is, half of the LoRA module, FedOTAB enjoys favorable communication efficiency. By explicitly splitting LoRA into local and global parts, FedOTAB effectively integrates both local and global knowledge. Extensive experiments demonstrate that FedOTAB excels in both homogeneous and heterogeneous rank scenarios, consistently outperforms state-of-the-art methods with remarkably low communication and computational overhead. We hope that this work provides valuable insights for federated LoRA fine-tuning, offering a practical and scalable solution to fine-tuning large models in mobile environments.
- Open Access
- 4 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 8-14
Perception-Aware Frame Display Scheduling for Low-Latency Cloud Gaming
- 4 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 8-14
Open AccessPerception-Aware Frame Display Scheduling for Low-Latency Cloud Gaming
- Open Access
Decoupling Defense Strategies for Robust Image Watermarking
Deep learning-based image watermarking, while robust against conventional distortions, remains vulnerable to advanced adversarial and regeneration attacks. Conventional countermeasures, which jointly optimize the encoder and decoder via a noise layer, face 2 inevitable challenges: (1) decrease of clean accuracy due to decoder adversarial training and (2) limited robustness due to simultaneous training of all three advanced attacks. To overcome these issues, we propose AdvMark, a novel two-stage fine-tuning framework that decouples the defense strategies. In stage 1, we address adversarial vulnerability via a tailored adversarial training paradigm that primarily fine-tunes the encoder while only conditionally updating the decoder. This approach learns to move the image into a non-attackable region, rather than modifying the decision boundary, thus preserving clean accuracy. In stage 2, we tackle distortion and regeneration attacks via direct image optimization. To preserve the adversarial robustness gained in stage 1, we formulate a principled, constrained image loss with theoretical guarantees, which balances the deviation from cover and previous encoded images. We also propose a quality-aware early-stop to further guarantee the lower bound of visual quality. Extensive experiments demonstrate AdvMark outperforms with the highest image quality and comprehensive robustness, i.e. up to 29\%, 33\% and 46\% accuracy improvement for distortion, regeneration and adversarial attacks, respectively.
Open AccessDecoupling Defense Strategies for Robust Image Watermarking
Deep learning-based image watermarking, while robust against conventional distortions, remains vulnerable to advanced adversarial and regeneration attacks. Conventional countermeasures, which jointly optimize the encoder and decoder via a noise layer, face 2 inevitable challenges: (1) decrease of clean accuracy due to decoder adversarial training and (2) limited robustness due to simultaneous training of all three advanced attacks. To overcome these issues, we propose AdvMark, a novel two-stage fine-tuning framework that decouples the defense strategies. In stage 1, we address adversarial vulnerability via a tailored adversarial training paradigm that primarily fine-tunes the encoder while only conditionally updating the decoder. This approach learns to move the image into a non-attackable region, rather than modifying the decision boundary, thus preserving clean accuracy. In stage 2, we tackle distortion and regeneration attacks via direct image optimization. To preserve the adversarial robustness gained in stage 1, we formulate a principled, constrained image loss with theoretical guarantees, which balances the deviation from cover and previous encoded images. We also propose a quality-aware early-stop to further guarantee the lower bound of visual quality. Extensive experiments demonstrate AdvMark outperforms with the highest image quality and comprehensive robustness, i.e. up to 29\%, 33\% and 46\% accuracy improvement for distortion, regeneration and adversarial attacks, respectively.
- Open Access
HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming
Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.
Open AccessHiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming
Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.
- Open Access
RRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations
With the growing demand for real-time video enhancement in live applications, existing methods often struggle to balance speed and effective exposure control, particularly under uneven lighting. We introduce RRNet (Rendering Relighting Network), a lightweight and configurable framework that achieves a state-of-the-art tradeoff between visual quality and efficiency. By estimating parameters for a minimal set of virtual light sources, RRNet enables localized relighting through a depth-aware rendering module without requiring pixel-aligned training data. This object-aware formulation preserves facial identity and supports real-time, high-resolution performance using a streamlined encoder and lightweight prediction head. To facilitate training, we propose a generative AI-based dataset creation pipeline that synthesizes diverse lighting conditions at low cost. With its interpretable lighting control and efficient architecture, RRNet is well suited for practical applications such as video conferencing, AR-based portrait enhancement, and mobile photography. Experiments show that RRNet consistently outperforms prior methods in low-light enhancement, localized illumination adjustment, and glare removal.
Open AccessRRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations
With the growing demand for real-time video enhancement in live applications, existing methods often struggle to balance speed and effective exposure control, particularly under uneven lighting. We introduce RRNet (Rendering Relighting Network), a lightweight and configurable framework that achieves a state-of-the-art tradeoff between visual quality and efficiency. By estimating parameters for a minimal set of virtual light sources, RRNet enables localized relighting through a depth-aware rendering module without requiring pixel-aligned training data. This object-aware formulation preserves facial identity and supports real-time, high-resolution performance using a streamlined encoder and lightweight prediction head. To facilitate training, we propose a generative AI-based dataset creation pipeline that synthesizes diverse lighting conditions at low cost. With its interpretable lighting control and efficient architecture, RRNet is well suited for practical applications such as video conferencing, AR-based portrait enhancement, and mobile photography. Experiments show that RRNet consistently outperforms prior methods in low-light enhancement, localized illumination adjustment, and glare removal.
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 12035-12044
Beyond Interpretability: Exploring the Comprehensibility of Adaptive Video Streaming through Large Language Models
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 12035-12044
Open AccessBeyond Interpretability: Exploring the Comprehensibility of Adaptive Video Streaming through Large Language Models
- Open Access
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 11844-11852
DynFed: Adaptive Federated Learning via Quantization-Aware Knowledge Distillation
- 27 October 2025
- Published by Association for Computing Machinery (ACM)
- p. 11844-11852
Open AccessDynFed: Adaptive Federated Learning via Quantization-Aware Knowledge Distillation
- Open Access
Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce Crucible, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate Crucible's effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that Crucible systematically quantifies the tunable space across different algorithms. Furthermore, Crucible provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.
Open AccessCrucible: Quantifying the Potential of Control Algorithms through LLM Agents
Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce Crucible, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate Crucible's effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that Crucible systematically quantifies the tunable space across different algorithms. Furthermore, Crucible provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.