Paper Info
Title
A Framework For Distributed Data-Parallel Execution In The Kepler Scientific Workflow System
Abstract
Distributed Data-Parallel (DDP) patterns such as MapReduce have become increasingly popular as solutions to facilitate data-intensive applications, resulting in a number of systems supporting DDP workflows. Yet, applications or workflows built using these patterns are usually tightly-coupled with the underlying DDP execution engine they select. We present a framework for distributed data-parallel execution in the Kepler scientific workflow system that enables users to easily switch between different DDP execution engines. We describe a set of DDP actors based on DDP patterns and directors for DDP workflow executions within the presented framework. We demonstrate how DDP workflows can be easily composed in the Kepler graphic user interface through the reuse of these DDP actors and directors and how the generated DDP workflows can be executed in different distributed environments. Via a bioinformatics usecase, we discuss the usability of the proposed framework and validate its execution scalability.
Year
DOI
Venue
2012
10.1016/j.procs.2012.04.178
PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE, ICCS 2012
Keywords
Field
DocType
Scientific workflows, distributed data-parallel patterns, data-intensive, bioinformatics
Data mining,Kepler scientific workflow system,Reuse,Computer science,Usability,Graphical user interface,Kepler,Scalability
Journal
Volume
ISSN
Citations 
9
1877-0509
6
PageRank 
References 
Authors
0.67
11
3
Name
Order
Citations
PageRank
Jianwu Wang121526.72
Daniel Crawl224321.02
Ilkay Altintas31191106.09