Paper Info
Title
Serverless execution of scientific workflows: Experiments with HyperFlow, AWS Lambda and Google Cloud Functions
Abstract
Scientific workflows consisting of a high number of interdependent tasks represent an important class of complex scientific applications. Recently, a new type of serverless infrastructures has emerged, represented by such services as Google Cloud Functions and AWS Lambda, also referred to as the Function-as-a-Service model. In this paper we take a look at such serverless infrastructures, which are designed mainly for processing background tasks of Web and Internet of Things applications, or event-driven stream processing. We evaluate their applicability to more compute- and data-intensive scientific workflows and discuss possible ways to repurpose serverless architectures for execution of scientific workflows. We have developed prototype workflow executor functions using AWS Lambda and Google Cloud Functions, coupled with the HyperFlow workflow engine. These functions can run workflow tasks in AWS and Google infrastructures, and feature such capabilities as data staging to/from S3 or Google Cloud Storage and execution of custom application binaries. We have successfully deployed and executed the Montage astronomy workflow, often used as a benchmark, and we report on initial results of its performance evaluation. Our findings indicate that the simple mode of operation makes this approach easy to use, although there are costs involved in preparing portable application binaries for execution in a remote environment.
Year
DOI
Venue
2020
10.1016/j.future.2017.10.029
Future Generation Computer Systems
Keywords
DocType
Volume
Scientific workflows,Cloud functions,Serverless architectures,FaaS
Journal
110
ISSN
Citations 
PageRank 
0167-739X
11
0.83
References 
Authors
13
5
Name
Order
Citations
PageRank
Maciej Malawski155346.80
Adam Gajek2171.84
Adam Zima3171.84
Bartosz Baliś4436.17
Kamil Figiela5906.20