Prediction of Optimal Parallelism Level in Wide Area Data Transfers
Authors:
- 15 September 2011
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Parallel and Distributed Systems
- Vol. 22 (12) , 2033-2045
Abstract
Wide area data transfer may be a major bottleneck for the end-to-end performance of distributed applications. A practical way of increasing the wide area throughput at the application layer is using multiple parallel streams. Although increased number of parallel streams may yield much better performance than using a single stream, overwhelming the network by opening too many streams may have an inverse effect. The congestion created by excess number of streams may cause a drop down in the throughput achieved. Hence, it is important to decide on the optimal number of streams without congesting the network. Predicting this "optimum” number is not straightforward, since it depends on many parameters specific to each individual transfer. Generic models that try to predict this number either rely too much on historical information or fail to achieve accurate predictions. In this paper, we present a set of new models which aim to approximate the optimal number with least history information and lowest prediction overhead. An algorithm is introduced to select the best combination of historic information to do the prediction for evaluation purposes as well as optimizing prediction by reducing error rate. We measure the feasibility and accuracy of the proposed prediction models by comparing to actual GridFTP data transfer by using little historical information and have seen that we could predict the throughput of parallel streams accurately and find a very close approximation of the optimal stream number.Keywords
This publication has 11 references indexed in Scilit:
- Target bandwidth sharing using endhost measuresPerformance Evaluation, 2007
- Parallel TCP Sockets: Simple Model, Throughput and Validation2006
- Adaptive data block scheduling for parallel TCP streams2005
- Characterizing and Predicting TCP Throughput on the Wide Area Network2005
- Modeling and Taming Parallel TCP on the Wide Area Network2005
- Stork: making data placement a first class citizen in the grid2004
- TCP behavior of a busy Internet server: analysis and improvements2002
- The end-to-end performance effects of parallel TCP sockets on a lossy wide-area network2002
- Effects of ensemble-TCPACM SIGCOMM Computer Communication Review, 2000
- Differentiated end-to-end Internet services using a weighted proportional fair sharing TCPACM SIGCOMM Computer Communication Review, 1998