Application-Level Optimization of Big Data Transfers through Pipelining, Parallelism and Concurrency
Authors:
- 23 March 2015
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Cloud Computing
- Vol. 4 (1) , 63-75
Abstract
In end-to-end data transfers, there are several factors affecting the data transfer throughput, such as the network characteristics (e.g., network bandwidth, round-trip-time, background traffic); end-system characteristics (e.g., NIC capacity, number of CPU cores and their clock rate, number of disk drives and their I/O rate); and the dataset characteristics (e.g., average file size, dataset size, file size distribution). Optimization of big data transfers over inter-cloud and intra-cloud networks is a challenging task that requires joint-consideration of all of these parameters. This optimization task becomes even more challenging when transferring datasets comprised of heterogeneous file sizes (i.e., large files and small files mixed). Previous work in this area only focuses on the end-system and network characteristics however does not provide models regarding the dataset characteristics. In this study, we analyze the effects of the three most important transfer parameters that are used to enhance data transfer throughput: pipelining,parallelism and concurrency. We provide models and guidelines to set the best values for these parameters and present two different transfer optimization algorithms that use the models developed. The tests conducted over high-speed networking and cloud testbeds show that our algorithms outperform the most popular data transfer tools like Globus Online and UDT in majority of the cases.Keywords
Funding Information
- National Science Foundation (NSF) (CNS-1131889, OCI-0926701, CCF-1115805)
This publication has 19 references indexed in Scilit:
- Dynamic Protocol Tuning Algorithms for High Performance Data Transfers2013
- End-to-End Data-Flow Parallelism for Throughput Optimization in High-Speed NetworksJournal of Grid Computing, 2012
- Prediction of Optimal Parallelism Level in Wide Area Data TransfersIEEE Transactions on Parallel and Distributed Systems, 2011
- Stork data scheduler: mitigating the data bottleneck in e-SciencePhilosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 2011
- Parallel TCP Sockets: Simple Model, Throughput and Validation2006
- Scalable socket buffer tuning for high-performance Web servers2005
- PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks2000
- Effects of ensemble-TCPACM SIGCOMM Computer Communication Review, 2000
- Differentiated end-to-end Internet services using a weighted proportional fair sharing TCPACM SIGCOMM Computer Communication Review, 1998
- The macroscopic behavior of the TCP congestion avoidance algorithmACM SIGCOMM Computer Communication Review, 1997