Paper Info
Title
A new paradigm for collaborating distributed query engines
Abstract
In this work we focus on using Distributed Caching Platform (DCP) to scale out database applications and to support relational data communication of multiple individual query-engines in a general graph-structured SQL dataflow process. While the use of DCP has gained popularity lately, transferring query results from one query-engine to another tuple-by-tuple through DCP is often inefficient; this is because the granularity of cache access is too small, and the overhead of data conversion and interpretation is too large. To deal with these issues, we leverage DCP's binary protocol and query engine's buffer management to deliver query results at the storage level directly. We extend the database buffer pool over multiple memory nodes to enable low-latency access to large volumes of data, and introduce the novel page-feed mechanism to allow the query results of the collaborative query engines to be communicated as data pages (blocks), namely, the producer query puts its result relation as pages in the DCP to be got by the consumer query. In this way, data are transferred as pages directly under DCP's binary protocol, where the contained tuples are exactly in the format required by the relational operators, and the use of pages, as mini-batches of tuples, provides the balanced efficiency of query processing and DCP access. Pushing relation data communication down to the storage (buffer pool) level from the application level offers significant performance gain, and is naturally consistent with the SQL semantics. We have implemented these specific mechanisms on a cluster of PostgreSQL engines. Our experiment results are documented in this paper.
Year
DOI
Venue
2012
10.1007/978-3-642-32584-7_34
DaWaK
Keywords
Field
DocType
data page,new paradigm,dcp access,data conversion,query result,query engine,producer query,query processing,collaborative query engine,consumer query,binary protocol
Query optimization,Web search query,Data mining,Query language,Query expansion,Computer science,Sargable,Web query classification,View,Query by Example,Database
Conference
Citations 
PageRank 
References 
0
0.34
4
Authors
2
Name
Order
Citations
PageRank
Qiming Chen12010233.16
Meichun Hsu23437778.34