Paper Info
Title
Coloc: Distributed Data And Container Colocation For Data-Intensive Applications
Abstract
The performance of scalable analytic frameworks supporting data-intensive parallel applications often depends significantly on the time it takes to read input data. Therefore, existing frameworks like Spark and Flink try to achieve a high degree of data locality by scheduling tasks on nodes where the input data resides. However, the set of nodes running a job and its tasks is chosen by a cluster resource management system like YARN, which schedules containers without taking the location of data into account. Yet, the scheduling of the frameworks is restricted to the set of nodes the containers are running on. At the same time, many jobs in productive clusters are recurring with predictable characteristics. For these jobs, it is possible to plan in advance on which nodes to place a job's input data and execution containers.In this paper we present CoLoc, a lightweight data and container scheduling assistant for recurring data-intensive analytic jobs. CoLoc allows users to define related files that serve as input for the same job. It colocates related files on a set of nodes and offers this scheduling hint to the cluster manager to also place the jobs container on these nodes. The main advantage of CoLoc is a reduction of network transfers due to a higher data locality and locally performed operators like grouping or joining two or more datasets. We implement CoLoc on Hadoop YARN and HDFS, then evaluate it on a 40 node cluster using workloads based on Apache Flink and the TPC-H benchmark suite. Compared to YARN's default scheduler and HDFS's block placement scheduler, CoLoc reduces the execution time up to 35% for the tested data-intensive workloads.
Year
Venue
Keywords
2016
2016 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA)
Resource Management, Data Placement, Parallel Dataflows, Scheduling, Data-Intensive Applications
Field
DocType
Citations 
Data mining,Locality,Spark (mathematics),Yarn,Scheduling (computing),Computer science,Schedule,Distributed database,Benchmark (computing),Scalability
Conference
0
PageRank 
References 
Authors
0.34
18
3
Name
Order
Citations
PageRank
Thomas Renner1185.47
Lauritz Thamsen2439.26
Odej Kao3106696.19