Paper Info
Title
SCOPE: parallel databases meet MapReduce
Abstract
Companies providing cloud-scale data services have increasing needs to store and analyze massive data sets, such as search logs, click streams, and web graph data. For cost and performance reasons, processing is typically done on large clusters of tens of thousands of commodity machines. Such massive data analysis on large clusters presents new opportunities and challenges for developing a highly scalable and efficient distributed computation system that is easy to program and supports complex system optimization to maximize performance and reliability. In this paper, we describe a distributed computation system, Structured Computations Optimized for Parallel Execution (Scope), targeted for this type of massive data analysis. Scope combines benefits from both traditional parallel databases and MapReduce execution engines to allow easy programmability and deliver massive scalability and high performance through advanced optimization. Similar to parallel databases, the system has a SQL-like declarative scripting language with no explicit parallelism, while being amenable to efficient parallel execution on large clusters. An optimizer is responsible for converting scripts into efficient execution plans for the distributed computation engine. A physical execution plan consists of a directed acyclic graph of vertices. Execution of the plan is orchestrated by a job manager that schedules execution on available machines and provides fault tolerance and recovery, much like MapReduce systems. Scope is being used daily for a variety of data analysis and data mining applications over tens of thousands of machines at Microsoft, powering Bing, and other online services.
Year
DOI
Venue
2012
10.1007/s00778-012-0280-z
The Vldb Journal
Keywords
DocType
Volume
SCOPE,Parallel databases,MapReduce,Distributed computation,Query optimization
Journal
21
Issue
ISSN
Citations 
5
1066-8888
56
PageRank 
References 
Authors
2.22
28
6
Name
Order
Citations
PageRank
Jingren Zhou182679.35
Nicolas Bruno256031.03
Ming-Chuan Wu328435.57
Per-åke Larson43262973.94
Ronnie Chaiken51343114.67
Darren Shakib645355.86