Lei Shi
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Lei Shi has a total of 15 co-authors. Here are the top co-authors by number of shared publications:
- Sen Liu
Fudan University, Shanghai, China
2 shared publications - Xuandong Liu
College of Computer Science and Artificial Intelligence, Fudan University, Shanghai, China
2 shared publications - Yang Xu
Fudan University, Shanghai, China
2 shared publications - Zixuan Chen
China Telecom, Shanghai, China
2 shared publications - Abdalkarim Awad
Birzeit University
1 shared publication - Falko Dressler
TU Berlin
1 shared publication
Latest Publications
- Open Access
- 7 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 102-111
OSP: Boosting Distributed Model Training with 2-stage Synchronization
Distributed deep learning (DDL) is a promising research area, which aims to increase the efficiency of training deep learning tasks with large size of datasets and models. As the computation capability of DDL nodes continues to increase, the network connection between nodes is becoming a major bottleneck. Various methods of gradient compression and improved model synchronization have been proposed to address this bottleneck in Parameter-Server-based DDL. However, these two types of methods can result in accuracy loss due to discarded gradients and have limited enhancement on the throughput of model synchronization, respectively. To address these challenges, we propose a new model synchronization method named Overlapped Synchronization Parallel (OSP), which achieves efficient communication with a 2-stage synchronization approach and uses Local-Gradient-based Parameter correction (LGP) to avoid accuracy loss caused by stale parameters. The prototype of OSP has been implemented using PyTorch and evaluated on commonly used deep learning models and datasets with a 9-node testbed. Evaluation results show that OSP can achieve up to 50% improvement in throughput without accuracy loss compared to popular synchronization models.
- 7 August 2023
- Published by Association for Computing Machinery (ACM)
- p. 102-111
Open AccessOSP: Boosting Distributed Model Training with 2-stage Synchronization
Distributed deep learning (DDL) is a promising research area, which aims to increase the efficiency of training deep learning tasks with large size of datasets and models. As the computation capability of DDL nodes continues to increase, the network connection between nodes is becoming a major bottleneck. Various methods of gradient compression and improved model synchronization have been proposed to address this bottleneck in Parameter-Server-based DDL. However, these two types of methods can result in accuracy loss due to discarded gradients and have limited enhancement on the throughput of model synchronization, respectively. To address these challenges, we propose a new model synchronization method named Overlapped Synchronization Parallel (OSP), which achieves efficient communication with a 2-stage synchronization approach and uses Local-Gradient-based Parameter correction (LGP) to avoid accuracy loss caused by stale parameters. The prototype of OSP has been implemented using PyTorch and evaluated on commonly used deep learning models and datasets with a 9-node testbed. Evaluation results show that OSP can achieve up to 50% improvement in throughput without accuracy loss compared to popular synchronization models.
- Open Access
- 28 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 65-66
Augmenting Distributed AI Training with Loss-tolerant Transmission
- 28 July 2023
- Published by Association for Computing Machinery (ACM)
- p. 65-66
Open AccessAugmenting Distributed AI Training with Loss-tolerant Transmission
- 1 February 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 111-118
Advantages of virtual addressing for efficient and failure tolerant routing in sensor networks
We study the capabilities of virtual addressing schemes for efficient and failure tolerant routing in sensor networks. In particular, we present the virtual cord protocol (VCP) that uses techniques known from peer-to-peer networks, i.e. distributed hash tables (DHTs) are used to associate data items in sensor networks with particular node addresses. The addresses of nodes are dynamically maintained by the protocol to form a virtual cord. VCP uses two mechanisms for finding paths to nodes and associated data items: First, it relies on the virtual cord that always points towards the destination. Furthermore, locally available neighborhood information is exploited for greedy routing. Our simulation results show that VCP is able to find paths close to the possible shortest path with very low overhead. The routing performance of VCP, which clearly outperforms other ad hoc routing protocols such as Dynamic MANET On Demand (DYMO), is similar to other virtual addressing schemes, e.g. Virtual Ring Routing (VRR). However, we improved VCP to handle frequent node failures in an optimized way. The results presented in this paper outline the capabilities of VCP to handle such cases.
- 1 February 2009
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 111-118
Advantages of virtual addressing for efficient and failure tolerant routing in sensor networks
We study the capabilities of virtual addressing schemes for efficient and failure tolerant routing in sensor networks. In particular, we present the virtual cord protocol (VCP) that uses techniques known from peer-to-peer networks, i.e. distributed hash tables (DHTs) are used to associate data items in sensor networks with particular node addresses. The addresses of nodes are dynamically maintained by the protocol to form a virtual cord. VCP uses two mechanisms for finding paths to nodes and associated data items: First, it relies on the virtual cord that always points towards the destination. Furthermore, locally available neighborhood information is exploited for greedy routing. Our simulation results show that VCP is able to find paths close to the possible shortest path with very low overhead. The routing performance of VCP, which clearly outperforms other ad hoc routing protocols such as Dynamic MANET On Demand (DYMO), is similar to other virtual addressing schemes, e.g. Virtual Ring Routing (VRR). However, we improved VCP to handle frequent node failures in an optimized way. The results presented in this paper outline the capabilities of VCP to handle such cases.
- 19 November 2007
- Published by SPIE-Intl Soc Optical Eng
- Vol. 6784, 678407
Static task scheduling based on ethernet virtual connections with varied granularities in ethernet over SDH networks
This paper describes the Task Scheduling problem over Ethernet Virtual Connections which provides multiple-granularity and the problem of scheduling tasks over two user network interfaces is considered. Three approximate models, 2SP, 2LSP and BWS are proposed with algorithms respectively. Numerical simulation results and comparisons are also presented.
- 19 November 2007
- Published by SPIE-Intl Soc Optical Eng
- Vol. 6784, 678407
Static task scheduling based on ethernet virtual connections with varied granularities in ethernet over SDH networks
This paper describes the Task Scheduling problem over Ethernet Virtual Connections which provides multiple-granularity and the problem of scheduling tasks over two user network interfaces is considered. Three approximate models, 2SP, 2LSP and BWS are proposed with algorithms respectively. Numerical simulation results and comparisons are also presented.