Efficient data transfer protocols for big data
Authors:
- 1 October 2012
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
- p. 1-9
Abstract
Data set sizes are growing exponentially, so it is important to use data movement protocols that are the most efficient available. Most data movement tools today rely on TCP over sockets, which limits flows to around 20Gbps on today's hardware. RDMA over Converged Ethernet (RoCE) is a promising new technology for high-performance network data movement with minimal CPU impact over circuit-based infrastructures. We compare the performance of TCP, UDP, UDT, and RoCE over high latency 10Gbps and 40Gbps network paths, and show that RoCE-based data transfers can fill a 40Gbps path using much less CPU than other protocols. We also show that the Linux zero-copy system calls can improve TCP performance considerably, especially on current Intel “Sandy Bridge”-based PCI Express 3.0 (Gen3) hosts.Keywords
This publication has 16 references indexed in Scilit:
- Evaluating High Performance Data Transfer with RDMA-based Protocols in Wide-Area Networks2012
- Efficient Zero-Copy Noncontiguous I/O for Globus on InfiniBand2010
- A comprehensive TCP fairness analysis in high speed networksComputer Communications, 2009
- Performance of HPC Middleware over InfiniBand WAN2008
- Data management and analysis for the Earth System GridJournal of Physics: Conference Series, 2008
- UDT: UDP-based data transfer for high-speed wide area networksComputer Networks, 2007
- HighSpeed TCP for Large Congestion Windows2003
- Research feature - SCTP: a proposed standard for robust internet data transportComputer, 2003
- Congestion control for high bandwidth-delay product networks2002
- TCP Extensions for High Performance1992