Rui Li
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Research Subjects
Rui Li has a total of 19 co-authors. Here are the top co-authors by number of shared publications:
- Xu Chen
Sun Yat-Sen University, Guangzhou, China
7 shared publications - Zhi Zhou
Sun Yat-sen University
5 shared publications - Jie Gong
School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China
2 shared publications - Qian Ma
School of Intelligent Systems Engineering, Sun Yat-sen University, Shenzhen, China
2 shared publications - Shuai Yu
Sun Yat-sen University
2 shared publications - Xiaoxi Zhang
Sun Yat-sen University
2 shared publications
Latest Publications
- Open Access
- 18 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 32944-32970
Efficient LLMs Training and Inference: An Introduction
ChatGPT was released in late November 2022, making a significant impact globally. Following this release, numerous domestic and international open-source projects for large model training emerged, including Alpaca, BOOLM, LLaMA, ChatGLM, DeepSpeedChat, and ColossalChat. Both academia and industry have a growing need to train large models for optimizing downstream tasks. Research has demonstrated that instruction tuning using high-quality training data can enable models with over 10 billion parameters to exhibit emergent abilities, particularly in complex reasoning tasks. Training a model with tens of billions of parameters is often necessary to improve business metrics using large model technology. However, most research teams lack the extensive computing resources required for such large-scale models, often possessing only a few V100 (32G) GPUs and occasionally a few A100s. Consequently, training or inferring a large model of this magnitude is impractical under limited computational resources. Therefore, adopting optimization strategies during the training and inference stages is essential to address these challenges. This survey summarizes a series of optimization techniques for large models during the training and inference phases, enabling the training of large models even with constrained computational resources.
- 18 November 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Access
- Vol. 13, 32944-32970
Open AccessEfficient LLMs Training and Inference: An Introduction
ChatGPT was released in late November 2022, making a significant impact globally. Following this release, numerous domestic and international open-source projects for large model training emerged, including Alpaca, BOOLM, LLaMA, ChatGLM, DeepSpeedChat, and ColossalChat. Both academia and industry have a growing need to train large models for optimizing downstream tasks. Research has demonstrated that instruction tuning using high-quality training data can enable models with over 10 billion parameters to exhibit emergent abilities, particularly in complex reasoning tasks. Training a model with tens of billions of parameters is often necessary to improve business metrics using large model technology. However, most research teams lack the extensive computing resources required for such large-scale models, often possessing only a few V100 (32G) GPUs and occasionally a few A100s. Consequently, training or inferring a large model of this magnitude is impractical under limited computational resources. Therefore, adopting optimization strategies during the training and inference stages is essential to address these challenges. This survey summarizes a series of optimization techniques for large models during the training and inference phases, enabling the training of large models even with constrained computational resources.
- 16 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Networking
- Vol. 32 (5) , 4414-4426
Online Optimization of DNN Inference Network Utility in Collaborative Edge Computing
Collaborative Edge Computing (CEC) is an emerging paradigm that collaborates heterogeneous edge devices as a resource pool to compute DNN inference tasks in proximity such as edge video analytics. Nevertheless, as the key knob to improve network utility in CEC, existing works mainly focus on the workload routing strategies among edge devices with the aim of minimizing the routing cost, remaining an open question for joint workload allocation and routing optimization problem from a system perspective. To this end, this paper presents a holistic, learned optimization for CEC towards maximizing the total network utility in an online manner, even though the utility functions of task input rates are unknown a priori. In particular, we characterize the CEC system in a flow model and formulate an online learning problem in a form of cross-layer optimization. We propose a nested-loop algorithm to solve workload allocation and distributed routing iteratively, using the tools of gradient sampling and online mirror descent. To improve the convergence rate over the nested-loop version, we further devise a single-loop algorithm. Rigorous analysis is provided to show its inherent convexity, efficient convergence, as well as algorithmic optimality. Finally, extensive numerical simulations demonstrate the superior performance of our solutions.
- 16 July 2024
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE/ACM Transactions on Networking
- Vol. 32 (5) , 4414-4426
Online Optimization of DNN Inference Network Utility in Collaborative Edge Computing
Collaborative Edge Computing (CEC) is an emerging paradigm that collaborates heterogeneous edge devices as a resource pool to compute DNN inference tasks in proximity such as edge video analytics. Nevertheless, as the key knob to improve network utility in CEC, existing works mainly focus on the workload routing strategies among edge devices with the aim of minimizing the routing cost, remaining an open question for joint workload allocation and routing optimization problem from a system perspective. To this end, this paper presents a holistic, learned optimization for CEC towards maximizing the total network utility in an online manner, even though the utility functions of task input rates are unknown a priori. In particular, we characterize the CEC system in a flow model and formulate an online learning problem in a form of cross-layer optimization. We propose a nested-loop algorithm to solve workload allocation and distributed routing iteratively, using the tools of gradient sampling and online mirror descent. To improve the convergence rate over the nested-loop version, we further devise a single-loop algorithm. Rigorous analysis is provided to show its inherent convexity, efficient convergence, as well as algorithmic optimality. Finally, extensive numerical simulations demonstrate the superior performance of our solutions.
- 14 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (5) , 5260-5279
FedLEO: An Offloading-Assisted Decentralized Federated Learning Framework for Low Earth Orbit Satellite Networks
Low Earth orbit (LEO) satellites enable complex Earth observation tasks (e.g., remote sensing and cooperative monitoring) by leveraging large-scale satellite-generated Earth imageries and state-of-the-art machine learning (ML) techniques. However, due to restricted downlink bandwidth and spotty connectivity, it is infeasible for the satellites to transmit all the imageries to ground stations for ML model training. To address this issue, we use federated learning (FL) to mitigate the significant overhead of raw data transmission only by enabling model parameter exchange. Traditional FL requires a central server for model parameter aggregation, which is impractical for distributed LEO satellite constellation due to the difficulty of identifying a suitable central satellite. To tackle such challenge, we take the unique topological characteristics of the LEO satellite constellation to design a decentralized FL framework that enables efficient model aggregation in LEO satellite networks without a central server. The framework can avoid the reliability and communication bandwidth problems of the central server in centralized FL. To mitigate the straggler effect and address the statistical heterogeneity, we then propose a novel offloading framework for decentralized FL in LEO satellite networks to aid the collaboration among multiple satellites for resource sharing. Based on it, we derive a satellite-centric threshold-based offloading strategy and a system-wide greedy-based iterative offloading decision making algorithm, in order to achieve delay and accuracy optimization under the computation and communication power constraints. Theoretical analysis demonstrates that the proposed framework contributes to the high training performance of the global model. Extensive experiments based on realistic datasets show that the proposed framework can reduce the system delay by up to 41% on average and improve the global model accuracy by up to 9.39% compared with benchmark policies.
- 14 August 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Mobile Computing
- Vol. 23 (5) , 5260-5279
FedLEO: An Offloading-Assisted Decentralized Federated Learning Framework for Low Earth Orbit Satellite Networks
Low Earth orbit (LEO) satellites enable complex Earth observation tasks (e.g., remote sensing and cooperative monitoring) by leveraging large-scale satellite-generated Earth imageries and state-of-the-art machine learning (ML) techniques. However, due to restricted downlink bandwidth and spotty connectivity, it is infeasible for the satellites to transmit all the imageries to ground stations for ML model training. To address this issue, we use federated learning (FL) to mitigate the significant overhead of raw data transmission only by enabling model parameter exchange. Traditional FL requires a central server for model parameter aggregation, which is impractical for distributed LEO satellite constellation due to the difficulty of identifying a suitable central satellite. To tackle such challenge, we take the unique topological characteristics of the LEO satellite constellation to design a decentralized FL framework that enables efficient model aggregation in LEO satellite networks without a central server. The framework can avoid the reliability and communication bandwidth problems of the central server in centralized FL. To mitigate the straggler effect and address the statistical heterogeneity, we then propose a novel offloading framework for decentralized FL in LEO satellite networks to aid the collaboration among multiple satellites for resource sharing. Based on it, we derive a satellite-centric threshold-based offloading strategy and a system-wide greedy-based iterative offloading decision making algorithm, in order to achieve delay and accuracy optimization under the computation and communication power constraints. Theoretical analysis demonstrates that the proposed framework contributes to the high training performance of the global model. Extensive experiments based on realistic datasets show that the proposed framework can reduce the system delay by up to 41% on average and improve the global model accuracy by up to 9.39% compared with benchmark policies.
- 7 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 10 (13) , 11453-11464
Reliability-Aware Online Scheduling for DNN Inference Tasks in Mobile-Edge Computing
Mobile-edge computing (MEC) is widely envisioned as a promising technique for provisioning artificial intelligence (AI) capability for resource-limited Internet of Things (IoT) devices by leveraging edge servers (ESs) for executing deep neural network (DNN) inference tasks in proximity. However, scheduling DNN inference tasks at the network edge under unknown system dynamics (e.g., uncertain availability of ESs) may suffer from failures, making it difficult to guarantee reliable services for the IoT device. To overcome this challenge, we propose a reliability-aware online scheduling scheme for DNN inference tasks in MEC by leveraging both online feedback and offline data to learn the uncertain availability of ESs to maximize both the inference accuracy and service reliability of DNN inference tasks (i.e., the number of DNN inference tasks processed during the system span). We first formulate the reliability-aware DNN inference tasks scheduling problem as a novel constrained combinatorial multiarmed bandit (CMAB) problem. Then by integrating the Lyapunov optimization technique, bandit learning, approximated submodular maximization, and historical data organically, we design a reliability-aware task scheduling scheme with a bandit learning (RTBL) algorithm to solve this problem. Unfortunately, even with an accurate prediction of the system uncertainties, the task scheduling problem is still NP-hard. To deal with it, we, therefore, design an advanced approximation algorithm based on the submodularity of the scheduling problem which obtains a near-optimal solution and provides a satisfactory performance guarantee. Finally, we conduct rigorous theoretical analysis and race-driven simulations to show RTBL’s brilliant performance.
- 7 February 2023
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 10 (13) , 11453-11464
Reliability-Aware Online Scheduling for DNN Inference Tasks in Mobile-Edge Computing
Mobile-edge computing (MEC) is widely envisioned as a promising technique for provisioning artificial intelligence (AI) capability for resource-limited Internet of Things (IoT) devices by leveraging edge servers (ESs) for executing deep neural network (DNN) inference tasks in proximity. However, scheduling DNN inference tasks at the network edge under unknown system dynamics (e.g., uncertain availability of ESs) may suffer from failures, making it difficult to guarantee reliable services for the IoT device. To overcome this challenge, we propose a reliability-aware online scheduling scheme for DNN inference tasks in MEC by leveraging both online feedback and offline data to learn the uncertain availability of ESs to maximize both the inference accuracy and service reliability of DNN inference tasks (i.e., the number of DNN inference tasks processed during the system span). We first formulate the reliability-aware DNN inference tasks scheduling problem as a novel constrained combinatorial multiarmed bandit (CMAB) problem. Then by integrating the Lyapunov optimization technique, bandit learning, approximated submodular maximization, and historical data organically, we design a reliability-aware task scheduling scheme with a bandit learning (RTBL) algorithm to solve this problem. Unfortunately, even with an accurate prediction of the system uncertainties, the task scheduling problem is still NP-hard. To deal with it, we, therefore, design an advanced approximation algorithm based on the submodularity of the scheduling problem which obtains a near-optimal solution and provides a satisfactory performance guarantee. Finally, we conduct rigorous theoretical analysis and race-driven simulations to show RTBL’s brilliant performance.
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (12) , 4581-4596
Joint Application Placement and Request Routing Optimization for Dynamic Edge Computing Service Management
As mobile edge computing (MEC) hosting applications at the network edge with limited capacities, service providers are facing the new challenge of how to make full use of the scarce edge resources to maximize the system performance. Accommodating this challenge requires careful application placement and request routing to coordinate diverse MEC nodes. However, frequent application re-placement would greatly increase the system reconfiguration cost, indicating a performance-cost trade-off. In response, in this paper, we study the problem of joint optimization on application placement and request routing to maximize the system performance, under a long-term budget of the application reconfiguration cost. Solving this problem is non-trivial since the long-term budget is coupled with the future system states (e.g., user request arrivals) that are typically unpredictable. To address this challenge, we first advocate an approximated dynamic optimization framework to decompose the long-term optimization problem into a series of one-shot problems which do not require the future system states. Moreover, since the decomposed problem is a mixed integer linear program (MILP) which is proven to be NP-hard, we then devise an efficient dependent rounding based approximation algorithm, which can achieve the near-optimal performance in a fast manner. Both rigorous theoretical analysis and extensive trace-driven evaluations demonstrate the proposed framework can achieve superior performance gain over existing schemes.
- 1 August 2022
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 33 (12) , 4581-4596
Joint Application Placement and Request Routing Optimization for Dynamic Edge Computing Service Management
As mobile edge computing (MEC) hosting applications at the network edge with limited capacities, service providers are facing the new challenge of how to make full use of the scarce edge resources to maximize the system performance. Accommodating this challenge requires careful application placement and request routing to coordinate diverse MEC nodes. However, frequent application re-placement would greatly increase the system reconfiguration cost, indicating a performance-cost trade-off. In response, in this paper, we study the problem of joint optimization on application placement and request routing to maximize the system performance, under a long-term budget of the application reconfiguration cost. Solving this problem is non-trivial since the long-term budget is coupled with the future system states (e.g., user request arrivals) that are typically unpredictable. To address this challenge, we first advocate an approximated dynamic optimization framework to decompose the long-term optimization problem into a series of one-shot problems which do not require the future system states. Moreover, since the decomposed problem is a mixed integer linear program (MILP) which is proven to be NP-hard, we then devise an efficient dependent rounding based approximation algorithm, which can achieve the near-optimal performance in a fast manner. Both rigorous theoretical analysis and extensive trace-driven evaluations demonstrate the proposed framework can achieve superior performance gain over existing schemes.
- 2 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Network Science and Engineering
- Vol. 8 (3) , 2288-2300
Dynamic Age Minimization With Real-Time Information Preprocessing for Edge-Assisted IoT Devices With Energy Harvesting
Age of information (AoI) is a newly proposed metric to quantify the freshness of system status. However, in many cases, the original raw data collected by IoT devices needs to be preprocessed in real-time to extract the hidden effective information, which is usually time consuming. To this end, we promote an edge computing assisted approach and aim to reduce the AoI by flexibly offloading the raw IoT data to the edge server for information preprocessing. We consider that the IoT devices can opportunistically collect extra energy through energy harvesting for sustainable operations, and propose a novel timely system status update model that consists of multiple IoT devices with energy harvesting and edge-assisted information preprocessing. The objective is to minimize the system-wide average AoI under a fixed energy cost budget. To tackle the key challenges due to the unpredictability of the stochastic energy harvesting process and the long-term energy constraints, we propose a Lyapunov-based average AoI Minimization (LAoIM) algorithm to derive an approximate optimal solution, and further quantify the performance gap from the optimal solution. Extensive numerical evaluations demonstrate that LAoIM can take full advantages of local and edge computation resources and achieve superior performance gain over existing schemes.
- 2 June 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Network Science and Engineering
- Vol. 8 (3) , 2288-2300
Dynamic Age Minimization With Real-Time Information Preprocessing for Edge-Assisted IoT Devices With Energy Harvesting
Age of information (AoI) is a newly proposed metric to quantify the freshness of system status. However, in many cases, the original raw data collected by IoT devices needs to be preprocessed in real-time to extract the hidden effective information, which is usually time consuming. To this end, we promote an edge computing assisted approach and aim to reduce the AoI by flexibly offloading the raw IoT data to the edge server for information preprocessing. We consider that the IoT devices can opportunistically collect extra energy through energy harvesting for sustainable operations, and propose a novel timely system status update model that consists of multiple IoT devices with energy harvesting and edge-assisted information preprocessing. The objective is to minimize the system-wide average AoI under a fixed energy cost budget. To tackle the key challenges due to the unpredictability of the stochastic energy harvesting process and the long-term energy constraints, we propose a Lyapunov-based average AoI Minimization (LAoIM) algorithm to derive an approximate optimal solution, and further quantify the performance gap from the optimal solution. Extensive numerical evaluations demonstrate that LAoIM can take full advantages of local and edge computation resources and achieve superior performance gain over existing schemes.
- 5 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 8 (19) , 14471-14484
Age of Processing: Age-Driven Status Sampling and Processing Offloading for Edge-Computing-Enabled Real-Time IoT Applications
The freshness of status information is of great importance for time-critical Internet-of-Things (IoT) applications. A metric measuring status freshness is the Age of Information (AoI), which captures the time elapsed from the status being generated at the source node (e.g., a sensor) to the latest status update. However, in intelligent IoT applications such as video surveillance, the status information is revealed after some computation-intensive and time-consuming data processing operations, which would affect the status freshness. In this article, we propose a novel metric, Age of Processing (AoP), to quantify such status freshness, which captures the time elapsed of the newest received processed status data since it is generated. Compared with AoI, AoP further takes the data processing time into account. Since an IoT device has limited computation and energy resources, the IoT device can choose to offload the data processing to the nearby edge server under constrained status sampling frequency. We aim to minimize the average AoP in a long-term process by jointly optimizing the status sampling frequency and processing offloading policy. We first formulate this online problem as an infinite-horizon constrained Markov decision process (CMDP) with an average reward criterion. We then transform the CMDP problem into an unconstrained Markov decision process (MDP) by leveraging a Lagrangian method, and accordingly propose a Lagrangian transformation framework for the original CMDP problem. Furthermore, we integrate the framework with a perturbation-based refinement mechanism for achieving the optimal policy of the CMDP problem. Our investigation shows that to minimize the average AoP: 1) for processing offloading: the policy exploits good channel state to offload processing to the edge server and 2) for status sampling: the waiting time presents a threshold structure. Extensive numerical evaluations show that the proposed algorithm outperforms the benchmarks, with an average AoP reduction up to 30%.
- 5 March 2021
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Internet of Things Journal
- Vol. 8 (19) , 14471-14484
Age of Processing: Age-Driven Status Sampling and Processing Offloading for Edge-Computing-Enabled Real-Time IoT Applications
The freshness of status information is of great importance for time-critical Internet-of-Things (IoT) applications. A metric measuring status freshness is the Age of Information (AoI), which captures the time elapsed from the status being generated at the source node (e.g., a sensor) to the latest status update. However, in intelligent IoT applications such as video surveillance, the status information is revealed after some computation-intensive and time-consuming data processing operations, which would affect the status freshness. In this article, we propose a novel metric, Age of Processing (AoP), to quantify such status freshness, which captures the time elapsed of the newest received processed status data since it is generated. Compared with AoI, AoP further takes the data processing time into account. Since an IoT device has limited computation and energy resources, the IoT device can choose to offload the data processing to the nearby edge server under constrained status sampling frequency. We aim to minimize the average AoP in a long-term process by jointly optimizing the status sampling frequency and processing offloading policy. We first formulate this online problem as an infinite-horizon constrained Markov decision process (CMDP) with an average reward criterion. We then transform the CMDP problem into an unconstrained Markov decision process (MDP) by leveraging a Lagrangian method, and accordingly propose a Lagrangian transformation framework for the original CMDP problem. Furthermore, we integrate the framework with a perturbation-based refinement mechanism for achieving the optimal policy of the CMDP problem. Our investigation shows that to minimize the average AoP: 1) for processing offloading: the policy exploits good channel state to offload processing to the edge server and 2) for status sampling: the waiting time presents a threshold structure. Extensive numerical evaluations show that the proposed algorithm outperforms the benchmarks, with an average AoP reduction up to 30%.
- 27 August 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 10 (1) , 648-661
Resource Price-Aware Offloading for Edge-Cloud Collaboration: A Two-Timescale Online Control Approach
Computation offloading is envisioned as a promising technique for prolonging the battery lives and enhancing the computation capability of mobile devices. In this paper, we study the task offloading and resource purchasing problems in an edge-cloud collaborative system. The purpose of this system is to minimize the cost of task offloading while ensuring that the tasks can be served before their maximum acceptable delays. Due to the uncertainty of both the task arrival rates and the prices of the computing resources, it is impossible to make an optimal decision online for a long-running time. Therefore, we propose a two-timescale Lyapunov optimization algorithm to overcome the uncertainty of the system’s future information and make the optimal decisions only based on the system’s current states. By purchasing computation resources in different timescales from the public cloud and making online decisions on where and how many requests should be offloaded, we can achieve an efficient outcome such that the system performance can approach the offline optimum without requiring a priori knowledge of system statistics. Rigorous theoretical analysis confirms the effectiveness of the proposed two-timescale Lyapunov optimization algorithm and extensive trace-driven experimental results show that the algorithm achieves outstanding performance gains over existing benchmarks.
- 27 August 2019
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 10 (1) , 648-661
Resource Price-Aware Offloading for Edge-Cloud Collaboration: A Two-Timescale Online Control Approach
Computation offloading is envisioned as a promising technique for prolonging the battery lives and enhancing the computation capability of mobile devices. In this paper, we study the task offloading and resource purchasing problems in an edge-cloud collaborative system. The purpose of this system is to minimize the cost of task offloading while ensuring that the tasks can be served before their maximum acceptable delays. Due to the uncertainty of both the task arrival rates and the prices of the computing resources, it is impossible to make an optimal decision online for a long-running time. Therefore, we propose a two-timescale Lyapunov optimization algorithm to overcome the uncertainty of the system’s future information and make the optimal decisions only based on the system’s current states. By purchasing computation resources in different timescales from the public cloud and making online decisions on where and how many requests should be offloaded, we can achieve an efficient outcome such that the system performance can approach the offline optimum without requiring a priori knowledge of system statistics. Rigorous theoretical analysis confirms the effectiveness of the proposed two-timescale Lyapunov optimization algorithm and extensive trace-driven experimental results show that the algorithm achieves outstanding performance gains over existing benchmarks.