Paper Info
Title
An Optimal Checkpointing Model with Online OCI Adjustment for Stream Processing Applications.
Abstract
Checkpoint-based fault-tolerant (FT) methods have been widely used to enhance the reliability of stream processing systems, but a checkpointing process usually introduces considerable overhead. It is a critical issue to choose the optimal checkpoint interval (OCI) that maximizes the processing efficiency. Traditional OCI models consider the recovery time equals to the execution time from the last checkpoint to the failure moment. However, for stream processing jobs, the recovery time is related to reprocessing workloads, depending on the real-time input data before a failure. A new model is needed to choose the OCI for stream processing applications. Moreover, the input data rate of a stream processing job fluctuates over time. To solve these problems, we present a novel DSPS OCI (DOCI) model in this paper. We prove that it maximizes the processing efficiency for a given time. We propose an approach to dynamically adjust the OCI for an application to accommodate the workload fluctuations. We conduct simulation experiments to verify the effectiveness of our DOCI model and the efficiency of the online OCI adjustment algorithm. Experimental results with a real-world dataset show that DOCI achieves an improvement on system efficiency by up to 32%, compared with existing FT approaches.
Year
DOI
Venue
2018
10.1002/cpe.5347
CONCURRENCY AND COMPUTATION-PRACTICE & EXPERIENCE
Keywords
Field
DocType
distributed stream processing,fault tolerance,optimal checkpoint interval
Task analysis,Workload,Computer science,Fault tolerance,Data rate,Execution time,Stream processing,Distributed computing
Conference
Volume
Issue
ISSN
31.0
20.0
1532-0626
Citations 
PageRank 
References 
0
0.34
0
Authors
5
Name
Order
Citations
PageRank
Yuan Zhuang165.84
Xiaohui Wei239154.44
Hongliang Li3192.73
Yongfang Wang4237.27
Xubin He574763.49