RDMA over Commodity Ethernet at Scale
Authors:
Open Access
- 22 August 2016
- conference paper
- Published by Association for Computing Machinery (ACM)
- p. 202-215
Abstract
Over the past one and half years, we have been using RDMA over commodity Ethernet (RoCEv2) to support some of Microsoft's highly-reliable, latency-sensitive services. This paper describes the challenges we encountered during the process and the solutions we devised to address them. In order to scale RoCEv2 beyond VLAN, we have designed a DSCP-based priority flow-control (PFC) mechanism to ensure large-scale deployment. We have addressed the safety challenges brought by PFC-induced deadlock (yes, it happened!), RDMA transport livelock, and the NIC PFC pause frame storm problem. We have also built the monitoring and management systems to make sure RDMA works as expected. Our experiences show that the safety and scalability issues of running RoCEv2 at scale can all be addressed, and RDMA can replace TCP for intra data center communications and achieve low latency, low CPU overhead, and high throughput.Keywords
This publication has 29 references indexed in Scilit:
- TIMELY2015
- Pingmesh2015
- Jupiter Rising2015
- Congestion Control for Large-Scale RDMA Deployments2015
- The tail at scaleCommunications of the ACM, 2013
- Improving datacenter performance and robustness with multipath TCPACM SIGCOMM Computer Communication Review, 2011
- BCubeACM SIGCOMM Computer Communication Review, 2009
- Prevention of deadlocks and livelocks in lossless backpressured packet networksIEEE/ACM Transactions on Networking, 2003
- Web search for a planet: the google cluster architectureIEEE Micro, 2003
- A necessary and sufficient condition for deadlock-free routing in cut-through and store-and-forward networksIEEE Transactions on Parallel and Distributed Systems, 1996