Shengbao Zheng
Current Organizations / Institutes
The organizations/institutes from the scholar's latest article
Previous Organizations / Institutes
The organizations / institutes from the scholar's other articles in the past ten years
Research Subjects
Shengbao Zheng has a total of 55 co-authors. Here are the top co-authors by number of shared publications:
- Adithya Gangidi
Meta, Menlo Park, USA
2 shared publications - Ashmitha Jeevaraj Shetty
Meta, Menlo Park, USA
2 shared publications - Hongyi Zeng
Meta, Menlo Park, USA
2 shared publications - Jingyi Yang
N/A, Menlo Park, USA
2 shared publications - Rui Miao
Meta, Menlo Park, USA
2 shared publications - Bingzhe Liu
Meta, Menlo Park, USA
1 shared publication
Latest Publications
- Open Access
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 505-518
Connecting 100K+ GPUs: Building the Communication Stack for Large-Scale LLM Training
- 11 August 2026
- Published by Association for Computing Machinery (ACM)
- p. 505-518
Open AccessConnecting 100K+ GPUs: Building the Communication Stack for Large-Scale LLM Training
- Top Cited PapersOpen Access
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 57-70
RDMA over Ethernet for Distributed Training at Meta Scale
The rapid growth in both computational density and scale in AI models in recent years motivates the construction of an efficient and reliable dedicated network infrastructure. This paper presents the design, implementation, and operation of Meta's Remote Direct Memory Access over Converged Ethernet (RoCE) networks for distributed AI training. Our design principles involve a deep understanding of the workloads, and we translated these insights into the design of various network components: Network Topology - To support the rapid evolution of generations of AI hardware platforms, we separated GPU-based training into its own "backend" network. Routing - Training workloads inherently impose load imbalance and burstiness, so we deployed several iterations of routing schemes to achieve near-optimal traffic distribution. Transport - We outline how we initially attempted to use DCQCN for congestion management but then pivoted away from DCQCN to instead leverage the collective library itself to manage congestion. Operations - We share our experience operating large-scale AI networks, including toolings we developed and troubleshooting examples.
- 4 August 2024
- Published by Association for Computing Machinery (ACM)
- p. 57-70
Top Cited PapersOpen AccessRDMA over Ethernet for Distributed Training at Meta Scale
The rapid growth in both computational density and scale in AI models in recent years motivates the construction of an efficient and reliable dedicated network infrastructure. This paper presents the design, implementation, and operation of Meta's Remote Direct Memory Access over Converged Ethernet (RoCE) networks for distributed AI training. Our design principles involve a deep understanding of the workloads, and we translated these insights into the design of various network components: Network Topology - To support the rapid evolution of generations of AI hardware platforms, we separated GPU-based training into its own "backend" network. Routing - Training workloads inherently impose load imbalance and burstiness, so we deployed several iterations of routing schemes to achieve near-optimal traffic distribution. Transport - We outline how we initially attempted to use DCQCN for congestion management but then pivoted away from DCQCN to instead leverage the collective library itself to manage congestion. Operations - We share our experience operating large-scale AI networks, including toolings we developed and troubleshooting examples.
- Open Access
- 17 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks
Building large AI fleets to support the rapidly growing DL workloads is an active research topic for modern cloud providers. Generating accurate benchmarks plays an essential role in designing the fast-paced software and hardware solutions in this space. Two fundamental challenges to make this scalable are (i) workload representativeness and (ii) the ability to quickly incorporate changes to the fleet into the benchmarks. To overcome these issues, we propose Mystique, an accurate and scalable framework for production AI benchmark generation. It leverages the PyTorch execution trace (ET), a new feature that captures the runtime information of AI models at the granularity of operators, in a graph format, together with their metadata. By sourcing fleet ETs, we can build AI benchmarks that are portable and representative. Mystique is scalable, due to its lightweight data collection, in terms of runtime overhead and instrumentation effort. It is also adaptive because ET composability allows flexible control on benchmark creation. We evaluate our methodology on several production AI models, and show that benchmarks generated with Mystique closely resemble original AI models, both in execution time and system-level metrics. We also showcase the portability of the generated benchmarks across platforms, and demonstrate several use cases enabled by the fine-grained composability of the execution trace.
- 17 June 2023
- Published by Association for Computing Machinery (ACM)
- p. 1-13
Open AccessMystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks
Building large AI fleets to support the rapidly growing DL workloads is an active research topic for modern cloud providers. Generating accurate benchmarks plays an essential role in designing the fast-paced software and hardware solutions in this space. Two fundamental challenges to make this scalable are (i) workload representativeness and (ii) the ability to quickly incorporate changes to the fleet into the benchmarks. To overcome these issues, we propose Mystique, an accurate and scalable framework for production AI benchmark generation. It leverages the PyTorch execution trace (ET), a new feature that captures the runtime information of AI models at the granularity of operators, in a graph format, together with their metadata. By sourcing fleet ETs, we can build AI benchmarks that are portable and representative. Mystique is scalable, due to its lightweight data collection, in terms of runtime overhead and instrumentation effort. It is also adaptive because ET composability allows flexible control on benchmark creation. We evaluate our methodology on several production AI models, and show that benchmarks generated with Mystique closely resemble original AI models, both in execution time and system-level metrics. We also showcase the portability of the generated benchmarks across platforms, and demonstrate several use cases enabled by the fine-grained composability of the execution trace.