Ruihao Gong
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Ruihao Gong has a total of 256 co-authors. Here are the top co-authors by number of shared publications:
- Xianglong Liu
Beihang University, Beijing, China
25 shared publications Xianglong Liu
State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing, China
19 shared publicationsFengwei Yu
SenseTime Research
18 shared publications- Jinyang Guo
Beihang University, Beijing, China
9 shared publications Yuhang Li
Department of Electrical Engineering, Yale University, New Haven, USA
9 shared publicationsHaotong Qin
ETH Zurich
8 shared publications
Latest Publications
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 18-25
TBQ:Trajectory-Aligned Branch-Aware Quantization for Few-Step MoE-Like Video Diffusion Models
High-quality video diffusion models are costly to deploy because they require long denoising trajectories while keeping large parameter footprints resident in memory. Reducing both the number of sampling steps and the numerical precision is therefore appealing, but these two compression axes interact in MoE video diffusion models: the quantizer must be calibrated on the distilled student trajectory, preserve the distinct statistics of the high-noise and low-noise experts, and accommodate phase-dependent activation ranges. We present TBQ, a trajectory-aligned, branch-aware compression pipeline for MoE-like video diffusion models. TBQ first distills the model into a few-step student and then calibrates low-bit quantization on the student’s own inference trajectory, maintaining separate expert-branch statistics and adopting a HiF4-style block floating-point format as the low-bit numerical representation. Evaluated on VBench across five dimensions—subject consistency, aesthetic quality, imaging quality, semantic alignment, and motion smoothness—the quantized student matches or improves upon the same-step full-precision model at moderate sampling budgets. At the representative 20-step setting, TBQ achieves an average VBench score of 0.7074, outperforming both the same-step full-precision model (0.7051) and the original full-precision baseline (0.6976), with subject consistency reaching 0.9629 and motion smoothness reaching 0.9825. These results demonstrate that trajectory-aligned, branch-aware calibration can make low-bit quantization a practical, quality-preserving component of memory- and latency-efficient video diffusion deployment.
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 18-25
Open AccessTBQ:Trajectory-Aligned Branch-Aware Quantization for Few-Step MoE-Like Video Diffusion Models
High-quality video diffusion models are costly to deploy because they require long denoising trajectories while keeping large parameter footprints resident in memory. Reducing both the number of sampling steps and the numerical precision is therefore appealing, but these two compression axes interact in MoE video diffusion models: the quantizer must be calibrated on the distilled student trajectory, preserve the distinct statistics of the high-noise and low-noise experts, and accommodate phase-dependent activation ranges. We present TBQ, a trajectory-aligned, branch-aware compression pipeline for MoE-like video diffusion models. TBQ first distills the model into a few-step student and then calibrates low-bit quantization on the student’s own inference trajectory, maintaining separate expert-branch statistics and adopting a HiF4-style block floating-point format as the low-bit numerical representation. Evaluated on VBench across five dimensions—subject consistency, aesthetic quality, imaging quality, semantic alignment, and motion smoothness—the quantized student matches or improves upon the same-step full-precision model at moderate sampling budgets. At the representative 20-step setting, TBQ achieves an average VBench score of 0.7074, outperforming both the same-step full-precision model (0.7051) and the original full-precision baseline (0.6976), with subject consistency reaching 0.9629 and motion smoothness reaching 0.9825. These results demonstrate that trajectory-aligned, branch-aware calibration can make low-bit quantization a practical, quality-preserving component of memory- and latency-efficient video diffusion deployment.
- Open Access
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8760-8769
OmniDTBC: Dynamic Token Budget Compression for Video-Audio Large Language Model
- 9 November 2026
- Published by Association for Computing Machinery (ACM)
- p. 8760-8769
Open AccessOmniDTBC: Dynamic Token Budget Compression for Video-Audio Large Language Model
- Open Access
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2356-2369
PiLLM: Resource-Efficient LLM Inference Using Workload Prediction
- 26 April 2026
- Published by Association for Computing Machinery (ACM)
- p. 2356-2369
Open AccessPiLLM: Resource-Efficient LLM Inference Using Workload Prediction
- Open Access
- 15 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 973-990
Towards Efficient LLM Inference via Collective and Adaptive Speculative Decoding
- 15 November 2025
- Published by Association for Computing Machinery (ACM)
- p. 973-990
Open AccessTowards Efficient LLM Inference via Collective and Adaptive Speculative Decoding
- 15 July 2025
- Published by Elsevier in Neural Networks
- Vol. 192, 107856
A survey of low-bit large language models: Basics, systems, and algorithms
- 15 July 2025
- Published by Elsevier in Neural Networks
- Vol. 192, 107856
A survey of low-bit large language models: Basics, systems, and algorithms
- Open Access
Past-Future Scheduler for LLM Serving under SLA Guarantees
The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service frameworks. The effectiveness of continuous batching relies on an accurate estimate of the memory requirements of requests. However, due to the diversity in request output lengths, existing frameworks tend to adopt aggressive or conservative schedulers, which often result in significant overestimation or underestimation of memory consumption. Consequently, they suffer from harmful request evictions or prolonged queuing times, failing to achieve satisfactory throughput under strict Service Level Agreement (SLA) guarantees (a.k.a. goodput), across various LLM application scenarios with differing input-output length distributions. To address this issue, we propose a novel Past-Future scheduler that precisely estimates the peak memory resources required by the running batch via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point. It adapts to applications with all types of input-output length distributions, balancing the trade-off between request queuing and harmful evictions, thereby consistently achieving better goodput. Furthermore, to validate the effectiveness of the proposed scheduler, we developed a high-performance LLM serving framework, LightLLM, that implements the Past-Future scheduler. Compared to existing aggressive or conservative schedulers, LightLLM demonstrates superior goodput, achieving up to 2-3$\times$ higher goodput than other schedulers under heavy loads. LightLLM is open source to boost the research in such direction (https://github.com/ModelTC/lightllm).
Open AccessPast-Future Scheduler for LLM Serving under SLA Guarantees
The exploration and application of Large Language Models (LLMs) is thriving. To reduce deployment costs, continuous batching has become an essential feature in current service frameworks. The effectiveness of continuous batching relies on an accurate estimate of the memory requirements of requests. However, due to the diversity in request output lengths, existing frameworks tend to adopt aggressive or conservative schedulers, which often result in significant overestimation or underestimation of memory consumption. Consequently, they suffer from harmful request evictions or prolonged queuing times, failing to achieve satisfactory throughput under strict Service Level Agreement (SLA) guarantees (a.k.a. goodput), across various LLM application scenarios with differing input-output length distributions. To address this issue, we propose a novel Past-Future scheduler that precisely estimates the peak memory resources required by the running batch via considering the historical distribution of request output lengths and calculating memory occupancy at each future time point. It adapts to applications with all types of input-output length distributions, balancing the trade-off between request queuing and harmful evictions, thereby consistently achieving better goodput. Furthermore, to validate the effectiveness of the proposed scheduler, we developed a high-performance LLM serving framework, LightLLM, that implements the Past-Future scheduler. Compared to existing aggressive or conservative schedulers, LightLLM demonstrates superior goodput, achieving up to 2-3$\times$ higher goodput than other schedulers under heavy loads. LightLLM is open source to boost the research in such direction (https://github.com/ModelTC/lightllm).
- 3 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-16
Temporal Feature Matters: A Framework for Diffusion Model Quantization
Diffusion models, widely used for image generation, face significant challenges related to their broad applicability due to prolonged inference times and high memory demands. Efficient Post-Training Quantization (PTQ) is crucial to address these issues. However, unlike traditional models, diffusion models critically rely on the timestep for the multi-round denoising. Typically, each timestep is encoded into a hypersensitive temporal feature by several modules. Despite this, existing PTQ methods do not optimize these modules individually. Instead, they employ unsuitable reconstruction objectives and complex calibration methods, leading to significant disturbances in the temporal feature and denoising trajectory, as well as reduced compression efficiency. To address these challenges, we introduce a novel quantization framework that includes three strategies: 1) TIB-based Maintenance: Based on our innovative Temporal Information Block (TIB) definition, Temporal Information-aware Reconstruction (TIAR) and Finite Set Calibration (FSC) are developed to efficiently align original temporal features. 2) Cache-based Maintenance: Instead of indirect and complex optimization for the related modules, pre-computing and caching quantized counterparts of temporal features are developed to minimize errors. 3) Disturbance-aware Selection: Employ temporal feature errors to guide a fine-grained selection between the two maintenance strategies for further disturbance reduction. This framework preserves most of the temporal information and ensures high-quality end-to-end generation. Extensive testing on various datasets, diffusion models, and hardware confirms our superior performance and acceleration.
- 3 July 2025
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Pattern Analysis and Machine Intelligence
- Vol. PP (01628828) , 1-16
Temporal Feature Matters: A Framework for Diffusion Model Quantization
Diffusion models, widely used for image generation, face significant challenges related to their broad applicability due to prolonged inference times and high memory demands. Efficient Post-Training Quantization (PTQ) is crucial to address these issues. However, unlike traditional models, diffusion models critically rely on the timestep for the multi-round denoising. Typically, each timestep is encoded into a hypersensitive temporal feature by several modules. Despite this, existing PTQ methods do not optimize these modules individually. Instead, they employ unsuitable reconstruction objectives and complex calibration methods, leading to significant disturbances in the temporal feature and denoising trajectory, as well as reduced compression efficiency. To address these challenges, we introduce a novel quantization framework that includes three strategies: 1) TIB-based Maintenance: Based on our innovative Temporal Information Block (TIB) definition, Temporal Information-aware Reconstruction (TIAR) and Finite Set Calibration (FSC) are developed to efficiently align original temporal features. 2) Cache-based Maintenance: Instead of indirect and complex optimization for the related modules, pre-computing and caching quantized counterparts of temporal features are developed to minimize errors. 3) Disturbance-aware Selection: Employ temporal feature errors to guide a fine-grained selection between the two maintenance strategies for further disturbance reduction. This framework preserves most of the temporal information and ensures high-quality end-to-end generation. Extensive testing on various datasets, diffusion models, and hardware confirms our superior performance and acceleration.
- Open Access
Post-Training Quantization for Video Matting
Video matting is crucial for applications such as film production and virtual reality, yet deploying its computationally intensive models on resource-constrained devices presents challenges. Quantization is a key technique for model compression and acceleration. As an efficient approach, Post-Training Quantization (PTQ) is still in its nascent stages for video matting, facing significant hurdles in maintaining accuracy and temporal coherence. To address these challenges, this paper proposes a novel and general PTQ framework specifically designed for video matting models, marking, to the best of our knowledge, the first systematic attempt in this domain. Our contributions include: (1) A two-stage PTQ strategy that combines block-reconstruction-based optimization for fast, stable initial quantization and local dependency capture, followed by a global calibration of quantization parameters to minimize accuracy loss. (2) A Statistically-Driven Global Affine Calibration (GAC) method that enables the network to compensate for cumulative statistical distortions arising from factors such as neglected BN layer effects, even reducing the error of existing PTQ methods on video matting tasks up to 20%. (3) An Optical Flow Assistance (OFA) component that leverages temporal and semantic priors from frames to guide the PTQ process, enhancing the model's ability to distinguish moving foregrounds in complex scenes and ultimately achieving near full-precision performance even under ultra-low-bit quantization. Comprehensive quantitative and visual results show that our PTQ4VM achieves the state-of-the-art accuracy performance across different bit-widths compared to the existing quantization methods. We highlight that the 4-bit PTQ4VM even achieves performance close to the full-precision counterpart while enjoying 8x FLOP savings.
Open AccessPost-Training Quantization for Video Matting
Video matting is crucial for applications such as film production and virtual reality, yet deploying its computationally intensive models on resource-constrained devices presents challenges. Quantization is a key technique for model compression and acceleration. As an efficient approach, Post-Training Quantization (PTQ) is still in its nascent stages for video matting, facing significant hurdles in maintaining accuracy and temporal coherence. To address these challenges, this paper proposes a novel and general PTQ framework specifically designed for video matting models, marking, to the best of our knowledge, the first systematic attempt in this domain. Our contributions include: (1) A two-stage PTQ strategy that combines block-reconstruction-based optimization for fast, stable initial quantization and local dependency capture, followed by a global calibration of quantization parameters to minimize accuracy loss. (2) A Statistically-Driven Global Affine Calibration (GAC) method that enables the network to compensate for cumulative statistical distortions arising from factors such as neglected BN layer effects, even reducing the error of existing PTQ methods on video matting tasks up to 20%. (3) An Optical Flow Assistance (OFA) component that leverages temporal and semantic priors from frames to guide the PTQ process, enhancing the model's ability to distinguish moving foregrounds in complex scenes and ultimately achieving near full-precision performance even under ultra-low-bit quantization. Comprehensive quantitative and visual results show that our PTQ4VM achieves the state-of-the-art accuracy performance across different bit-widths compared to the existing quantization methods. We highlight that the 4-bit PTQ4VM even achieves performance close to the full-precision counterpart while enjoying 8x FLOP savings.
- Open Access
Pre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
Extensive LLM applications demand efficient structured generations, particularly for LR(1) grammars, to produce outputs in specified formats (e.g., JSON). Existing methods primarily parse LR(1) grammars into a pushdown automaton (PDA), leading to runtime execution overhead for context-dependent token processing, especially inefficient under large inference batches. To address these issues, we propose Pre$^3$ that exploits deterministic pushdown automata (DPDA) to optimize the constrained LLM decoding efficiency. First, by precomputing prefix-conditioned edges during the preprocessing, Pre$^3$ enables ahead-of-time edge analysis and thus makes parallel transition processing possible. Second, by leveraging the prefix-conditioned edges, Pre$^3$ introduces a novel approach that transforms LR(1) transition graphs into DPDA, eliminating the need for runtime path exploration and achieving edge transitions with minimal overhead. Pre$^3$ can be seamlessly integrated into standard LLM inference frameworks, reducing time per output token (TPOT) by up to 40% and increasing throughput by up to 36% in our experiments. Our code is available at https://github.com/ModelTC/lightllm.
Open AccessPre$^3$: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
Extensive LLM applications demand efficient structured generations, particularly for LR(1) grammars, to produce outputs in specified formats (e.g., JSON). Existing methods primarily parse LR(1) grammars into a pushdown automaton (PDA), leading to runtime execution overhead for context-dependent token processing, especially inefficient under large inference batches. To address these issues, we propose Pre$^3$ that exploits deterministic pushdown automata (DPDA) to optimize the constrained LLM decoding efficiency. First, by precomputing prefix-conditioned edges during the preprocessing, Pre$^3$ enables ahead-of-time edge analysis and thus makes parallel transition processing possible. Second, by leveraging the prefix-conditioned edges, Pre$^3$ introduces a novel approach that transforms LR(1) transition graphs into DPDA, eliminating the need for runtime path exploration and achieving edge transitions with minimal overhead. Pre$^3$ can be seamlessly integrated into standard LLM inference frameworks, reducing time per output token (TPOT) by up to 40% and increasing throughput by up to 36% in our experiments. Our code is available at https://github.com/ModelTC/lightllm.
- Open Access
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (15) , 16354-16362
AtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints
Tiny machine learning (TinyML) has attracted heightened attention for its ability to provide low-cost and instantaneous performance on edge devices. Particularly, the commonly used microcontroller unit (MCU) imposes extreme constraints on peak memory (SRAM) and storage (Flash). Existing TinyML methods often rely on a customized and hard-to-obtain inference libraries, as well as necessitate a time-consuming search for a deployable architecture using advanced Neural Architecture Search (NAS) algorithms. To solve these problems, we fully exploit the resources on MCU and deduce hardware-oriented guidelines for designing models under extreme MCU constraints. In detail, we delve into thorough information about the atom operators by collecting the runtime data of Flash, SRAM, and latency to build a dataset named AtomDB. Based on AtomDB, several critical operator guidelines are established to fully utilize limited Flash and SRAM, while minimizing latency. By transferring the guidelines to analyze blocks, we propose a hybrid pattern that organizes appropriate blocks at different network stages to form the AtomNet, a more hardware-oriented architecture, to handle the former SRAM bottleneck and the latter Flash bottleneck. Extensive experiments demonstrate the effectiveness of the exploitation of the hardware characteristics. Remarkably, AtomNet pioneeringly achieve 3.5% accuracy enhancement and more than 15% latency reduction on 320KB MCU using readily available official inference libraries for ImageNet tasks, surpassing the current state-of-the-art method.
- 11 April 2025
- Published by Association for the Advancement of Artificial Intelligence (AAAI) in Proceedings of the AAAI Conference on Artificial Intelligence
- Vol. 39 (15) , 16354-16362
Open AccessAtomNet: Designing Tiny Models from Operators Under Extreme MCU Constraints
Tiny machine learning (TinyML) has attracted heightened attention for its ability to provide low-cost and instantaneous performance on edge devices. Particularly, the commonly used microcontroller unit (MCU) imposes extreme constraints on peak memory (SRAM) and storage (Flash). Existing TinyML methods often rely on a customized and hard-to-obtain inference libraries, as well as necessitate a time-consuming search for a deployable architecture using advanced Neural Architecture Search (NAS) algorithms. To solve these problems, we fully exploit the resources on MCU and deduce hardware-oriented guidelines for designing models under extreme MCU constraints. In detail, we delve into thorough information about the atom operators by collecting the runtime data of Flash, SRAM, and latency to build a dataset named AtomDB. Based on AtomDB, several critical operator guidelines are established to fully utilize limited Flash and SRAM, while minimizing latency. By transferring the guidelines to analyze blocks, we propose a hybrid pattern that organizes appropriate blocks at different network stages to form the AtomNet, a more hardware-oriented architecture, to handle the former SRAM bottleneck and the latter Flash bottleneck. Extensive experiments demonstrate the effectiveness of the exploitation of the hardware characteristics. Remarkably, AtomNet pioneeringly achieve 3.5% accuracy enhancement and more than 15% latency reduction on 320KB MCU using readily available official inference libraries for ImageNet tasks, surpassing the current state-of-the-art method.