Paper Info
Title
Efficient and Adaptive Stateful Replication for Stream Processing Engines in High-Availability Cluster
Abstract
Stateful stream process engines in high availability clusters (HACs) track a large number of concurrent flow states and replicate them to backups to provide reliable functionality. Under high traffic loads, existing solutions in such HACs are expensive owing to precise stateful replication. This work presents two novel methods to address this issue: randomization on replication representation and a replication scheme designed for when system becomes overloaded. A hashing structure called Multilevel Counting Bloom Filter (MLCBF) is proposed as a low resource-consuming solution of stateful replication. Its performance and tradeoffs are then evaluated based on theoretic analysis and extensive trace-based tests. Trace-based simulation reveals that MLCBF reduces network and memory requirements of replication typically by over 90 percent for URL categorization. Most importantly, MLCBF is quite as simple and practical for implementation and maintenance. Moreover, an adaptive scheme called dynamic lazy insertion is designed to prevent replication from overloading system continuously and optimize the throughput of HAC. Testbed evaluation demonstrates its feasibility and effectiveness in an overloaded HAC.
Year
DOI
Venue
2011
10.1109/TPDS.2011.83
IEEE Trans. Parallel Distrib. Syst.
Keywords
Field
DocType
replication scheme,bloom filter,stateful replication,high traffic load,overloaded hac,adaptive scheme,adaptive stateful replication,overloading system,stream processing engines,precise stateful replication,high availability cluster,replication representation,high-availability cluster,parallel processing,bloom filters,hash function,replication,stream processing,data structures,high availability
Bloom filter,High-availability cluster,Computer science,Testbed,Real-time computing,Hash function,Stateful firewall,Throughput,Stream processing,High availability,Distributed computing
Journal
Volume
Issue
ISSN
22
11
1045-9219
Citations 
PageRank 
References 
3
0.47
16
Authors
3
Name
Order
Citations
PageRank
Yi-hsuan Feng1112.54
Nen-Fu Huang262072.93
Yen-min Wu341.53