Paper Info
Title
A data transfer framework for large-scale science experiments
Abstract
Modern scientific experiments can generate hundreds of gigabytes to terabytes or even petabytes of data that may furthermore be maintained in large numbers of relatively small files. Frequently, this data must be disseminated to remote collaborators or computational centers for data analysis. Moving this data with high performance and strong robustness and providing a simple interface for users are challenging tasks. We present a data transfer framework comprising a high-performance data transfer library based on GridFTP, a data scheduler, and a graphical user interface that allows users to transfer their data easily, reliably, and securely. This system incorporates automatic tuning mechanisms to select at runtime the number of concurrent threads to be used for transfers. Also included are restart mechanisms capable of dealing with client, network, and server failures. Experimental results indicate that our data transfer system can significantly improve data transfer performance and can recover well from failures.
Year
DOI
Venue
2010
10.1145/1851476.1851582
HPDC
Keywords
Field
DocType
high-performance data transfer library,data analysis,large-scale science experiment,data scheduler,simple interface,data transfer performance,data transfer framework,automatic tuning mechanism,high performance,data transfer system,graphical user interface,graphic user interface,data transfer
Data transmission,Computer science,Terabyte,Petabyte,Gigabyte,Thread (computing),Robustness (computer science),Real-time computing,Graphical user interface,GridFTP,Distributed computing
Conference
Citations 
PageRank 
References 
29
1.06
9
Authors
4
Name
Order
Citations
PageRank
Wantao Liu1738.29
Brian Tieman2579.77
Rajkumar Kettimuthu377070.13
Foster Ian4229382663.24