MRTOM: Mostly Reliable Totally Ordered Multicast, a Network Primitive to Offload Distributed Systems
Authors:
- 1 July 2023
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 638-648
Abstract
As datacenters become the new computing platform, integrating server-centric distributed systems into modern network hardware is gaining interest under the diminishing Moore's law. Researchers want to build reusable primitives that can take advantage of modern network hardware and offload common system components of a broad range of applications. In this paper, we present Mostly Reliable Totally Ordered Multicast, a reusable network primitive that can embed reliable group communication into the network. MRTOM is a network-centric approach that handles message replication, ordering, and reliable delivery using a network fast path, freeing server CPUs for application logic. MRTOM can be implemented in the programmable switches and edge interfaces (e.g., Smart-NICs), significantly reducing network traffic compared to the existing approaches and improving job finish time amid packet loss. With MRTOM, we were able to accelerate multiple high-performance applications whose fast path can be totally offloaded into the network. For example, a Paxos application, MRTOM-Paxos, achieves > 1,100,000 transactions/secs and $23\mu \mathrm{s}$ minimum latency. A replicated key-value store, MRTOM-KV, also shows significant latency reduction with eBPF/XDP in the Linux Kernel, which is further improved by offloading into SmartNICs.
Keywords
Funding Information
- NSF (1705095)
This publication has 28 references indexed in Scilit:
- Jupiter RisingACM SIGCOMM Computer Communication Review, 2015
- NetPaxos2015
- Raft RefloatedACM SIGOPS Operating Systems Review, 2015
- CORFUACM Transactions on Computer Systems, 2013
- Forwarding metamorphosisACM SIGCOMM Computer Communication Review, 2013
- Fast PaxosDistributed Computing, 2006
- Total order multicast to multiple groups2002
- The part-time parliamentACM Transactions on Computer Systems, 1998
- Ordered and reliable multicast communicationACM Transactions on Computer Systems, 1991
- Consensus in the presence of partial synchronyJournal of the ACM, 1988