Paper Info
Title
Qbdj: A Novel Framework For Handling Skew In Parallel Join Processing On Distributed Memory
Abstract
The performance of parallel distributed data management systems becomes increasingly important with the rise of Big Data. Parallel joins have been widely studied both in the parallel processing and the database communities. Nevertheless, most of the algorithms so far developed do not consider the data skew, which naturally exists in various applications. State of the art methods designed to handle this problem are based on extensions to either of the two prevalent conventional approaches to parallel joins - the hash-based and duplication-based frameworks. In this paper, we introduce a novel parallel join framework, query-based distributed join (QbDJ), for handling data skew on distributed architectures. Further, we present an efficient implementation of the method based on the asynchronous partitioned global address space (APGAS) parallel programming model. We evaluate the performance of our approach on a cluster of 192 cores (16 nodes) and datasets of 1 billion tuples with different skews. The results show that the method is scalable, and also runs faster with less network communication compared to state-of-art PRPD approach in [1] under high data skew.
Year
DOI
Venue
2013
10.1109/HPCC.and.EUC.2013.214
2013 IEEE 15TH INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING AND COMMUNICATIONS & 2013 IEEE INTERNATIONAL CONFERENCE ON EMBEDDED AND UBIQUITOUS COMPUTING (HPCC_EUC)
Keywords
Field
DocType
Distributed join, parallel join, data skew, high performance, X10
Joins,Computer science,Tuple,Parallel computing,Distributed memory,Parallel programming model,Skew,Distributed database,Partitioned global address space,Scalability,Distributed computing
Conference
Citations 
PageRank 
References 
7
0.51
12
Authors
4
Name
Order
Citations
PageRank
Long Cheng19116.99
Spyros Kotoulas259046.46
Tomas E. Ward310419.10
Georgios Theodoropoulos433231.39