LSLifeng Sun

Lifeng Sun

*This is an algorithmically generated author record based on publicly available information.

Current Organizations / Institutes

The organizations/institutes from the scholar's latest article


Lifeng Sun has a total of 114 co-authors. Here are the top co-authors by number of shared publications:


Latest Publications

  • Open Access

    FedICL-KVc: Federated In-Context Learning via KV Cache

    by Yiming Chen, Nan He, Zheng Jiang, Lifeng Sun

  • Federated LoRA Fine-Tuning of LLMs with Only Transmitting Matrix A or B

    by Yiming Chen, Ruanjun Li, Jiawei Shao, Lifeng Sun, Chi Zhang, Jun Zhang, Xuelong Li

    Recent advancements in large language models (LLMs) have been largely driven by vast amounts of training data. As public data are exhausted, fine-tuning LLMs with private data has become increasingly important. However, due to privacy concerns associated with sensitive data, traditional centralized training approaches are not feasible. Federated learning (FL) offers a promising solution, as it enables collaborative model training across multiple mobile devices without data exchange. Nevertheless, it is challenging to fine-tune LLMs with FL on resource-constrained and heterogeneous clients. To reduce the cost, we employ Low-Rank Adaptation (LoRA) for efficient fine-tuning. Existing federated LoRA fine-tuning methods suffer from inaccurate aggregation, and the resulting noise may lead to performance loss that may be more severe for LLMs. In this paper, we propose a novel method, FedOTAB, to mitigate this issue. FedOTAB first correctly aggregates all local LoRA modules and then fixes the matrix $\bf {A}$ or $\bf {B}$ to optimize the corresponding best approximate matrix $\bf {B}^*$ or $\bf {A}^*$ for each local LoRA module by minimizing the induced noise. In addition, we design a mechanism to freeze the matrices $\bf {A}$ and $\bf {B}$ alternately to stabilize optimization. By transmitting only the updated part, that is, half of the LoRA module, FedOTAB enjoys favorable communication efficiency. By explicitly splitting LoRA into local and global parts, FedOTAB effectively integrates both local and global knowledge. Extensive experiments demonstrate that FedOTAB excels in both homogeneous and heterogeneous rank scenarios, consistently outperforms state-of-the-art methods with remarkably low communication and computational overhead. We hope that this work provides valuable insights for federated LoRA fine-tuning, offering a practical and scalable solution to fine-tuning large models in mobile environments.


  • Open Access

    Decoupling Defense Strategies for Robust Image Watermarking

    by Jiahui Chen, Zehang Deng, Zeyu Zhang, Chaoyang Li, Lianchen Jia, Lifeng Sun

    Deep learning-based image watermarking, while robust against conventional distortions, remains vulnerable to advanced adversarial and regeneration attacks. Conventional countermeasures, which jointly optimize the encoder and decoder via a noise layer, face 2 inevitable challenges: (1) decrease of clean accuracy due to decoder adversarial training and (2) limited robustness due to simultaneous training of all three advanced attacks. To overcome these issues, we propose AdvMark, a novel two-stage fine-tuning framework that decouples the defense strategies. In stage 1, we address adversarial vulnerability via a tailored adversarial training paradigm that primarily fine-tunes the encoder while only conditionally updating the decoder. This approach learns to move the image into a non-attackable region, rather than modifying the decision boundary, thus preserving clean accuracy. In stage 2, we tackle distortion and regeneration attacks via direct image optimization. To preserve the adversarial robustness gained in stage 1, we formulate a principled, constrained image loss with theoretical guarantees, which balances the deviation from cover and previous encoded images. We also propose a quality-aware early-stop to further guarantee the lower bound of visual quality. Extensive experiments demonstrate AdvMark outperforms with the highest image quality and comprehensive robustness, i.e. up to 29\%, 33\% and 46\% accuracy improvement for distortion, regeneration and adversarial attacks, respectively.

  • Open Access

    HiVid: LLM-Guided Video Saliency For Content-Aware VOD And Live Streaming

    by Jiahui Chen, Bo Peng, Lianchen Jia, Zeyu Zhang, Tianchi Huang, Lifeng Sun

    Content-aware streaming requires dynamic, chunk-level importance weights to optimize subjective quality of experience (QoE). However, direct human annotation is prohibitively expensive while vision-saliency models generalize poorly. We introduce HiVid, the first framework to leverage Large Language Models (LLMs) as a scalable human proxy to generate high-fidelity weights for both Video-on-Demand (VOD) and live streaming. We address 3 non-trivial challenges: (1) To extend LLMs' limited modality and circumvent token limits, we propose a perception module to assess frames in a local context window, autoregressively building a coherent understanding of the video. (2) For VOD with rating inconsistency across local windows, we propose a ranking module to perform global re-ranking with a novel LLM-guided merge-sort algorithm. (3) For live streaming which requires low-latency, online inference without future knowledge, we propose a prediction module to predict future weights with a multi-modal time series model, which comprises a content-aware attention and adaptive horizon to accommodate asynchronous LLM inference. Extensive experiments show HiVid improves weight prediction accuracy by up to 11.5\% for VOD and 26\% for live streaming over SOTA baselines. Real-world user study validates HiVid boosts streaming QoE correlation by 14.7\%.


  • Open Access

    RRNet: Configurable Real-Time Video Enhancement with Arbitrary Local Lighting Variations

    by Wenlong Yang, Canran Jin, Weihang Yuan, Chao Wang, Lifeng Sun

    With the growing demand for real-time video enhancement in live applications, existing methods often struggle to balance speed and effective exposure control, particularly under uneven lighting. We introduce RRNet (Rendering Relighting Network), a lightweight and configurable framework that achieves a state-of-the-art tradeoff between visual quality and efficiency. By estimating parameters for a minimal set of virtual light sources, RRNet enables localized relighting through a depth-aware rendering module without requiring pixel-aligned training data. This object-aware formulation preserves facial identity and supports real-time, high-resolution performance using a streamlined encoder and lightweight prediction head. To facilitate training, we propose a generative AI-based dataset creation pipeline that synthesizes diverse lighting conditions at low cost. With its interpretable lighting control and efficient architecture, RRNet is well suited for practical applications such as video conferencing, AR-based portrait enhancement, and mobile photography. Experiments show that RRNet consistently outperforms prior methods in low-light enhancement, localized illumination adjustment, and glare removal.

  • Open Access

    Beyond Interpretability: Exploring the Comprehensibility of Adaptive Video Streaming through Large Language Models

    by Lianchen Jia, Chaoyang Li, Ziqi Yuan, Jiahui Chen, Tianchi Huang, Jiangchuan Liu, Lifeng Sun


  • Open Access

    Crucible: Quantifying the Potential of Control Algorithms through LLM Agents

    by Lianchen Jia, Chaoyang Li, Qian Houde, Tianchi Huang, Jiangchuan Liu, Lifeng Sun

    Control algorithms in production environments typically require domain experts to tune their parameters and logic for specific scenarios. However, existing research predominantly focuses on algorithmic performance under ideal or default configurations, overlooking the critical aspect of Tuning Potential. To bridge this gap, we introduce Crucible, an agent that employs an LLM-driven, multi-level expert simulation to turn algorithms and defines a formalized metric to quantitatively evaluate their Tuning Potential. We demonstrate Crucible's effectiveness across a wide spectrum of case studies, from classic control tasks to complex computer systems, and validate its findings in a real-world deployment. Our experimental results reveal that Crucible systematically quantifies the tunable space across different algorithms. Furthermore, Crucible provides a new dimension for algorithm analysis and design, which ultimately leads to performance improvements. Our code is available at https://github.com/thu-media/Crucible.