Paper Info
Title
A Data-Aware Partitioning and Optimization Method for Large-Scale Workflows in Hybrid Computing Environments
Abstract
While hybrid computing environments provide good potential for achieving high performance and low economic cost, it also introduces a broad set of unpredictable overheads especially for running data-intensive applications. This paper describes a novel approach which refines workflow structures and optimizes intermediate data transfers for large-scale scientific workflows containing thousands (or even millions) of tasks. The proposed method includes pre- and post-partitioning of workflows and data-flow optimization. Firstly, it partitions a workflow by identifying the critical path of the task graph. Secondly, it controls the granularity of partitions to reduce the complexity of task graph in order to process large-scale workflows. Thirdly, it optimizes the data-flow based on the scheduling to minimize its communication overheads. Our proposed approach is able to handle complex data flows and significantly reduce data transfer by replacing individual tasks according to data dependencies. We conducted experiments using real applications such as Montage and Broadband, and the results demonstrated the effectiveness of our methods in achieving low execution time with low communication overhead in a hybrid computing environments.
Year
DOI
Venue
2013
10.1109/ICPADS.2013.29
ICPADS
Keywords
Field
DocType
optimisation,data dependency,grid computing,low economic cost,data-flow optimization,hybrid computing environment,hybrid computing environments,large-scale scientific workflow,complex data flow,low communication overhead,optimization method,graph theory,data-aware partitioning,task graph,large-scale scientific workflows,large-scale workflows,optimizes intermediate data transfer,data transfer
Graph theory,Algorithm design,Grid computing,Data transmission,Computer science,Scheduling (computing),Critical path method,Granularity,Workflow,Distributed computing
Conference
ISSN
Citations 
PageRank 
1521-9097
0
0.34
References 
Authors
5
2
Name
Order
Citations
PageRank
Rubing Duan123516.38
Xiaorong Li2473.62