Name
Abstract | ||
|---|---|---|
Stream Processing (SP) is now a major paradigm to timely handle large volumes of data generated at the edge of the Internet. Stream processing engines (SPE) are tools easing the specification, deployment and monitoring of SP applications. Such applications are typically programmed as a directed acyclic graph (DAG) of operators to be applied on each data item. Yet, SPEs are mostly equipped to deploy one application at a time without seeking synergies between those applications. Yet, in many domains, the set of operators composing applications overlap for a non-negligible amount. We envision a platform on which applications are submitted dynamically, each new graph of operators potentially sharing some of them with the currently running operators. We assume a homogeneous platform, a graph being deployed over multiple nodes. We need to minimize the inter-node traffic while guaranteeing that the capacity of a node is not exceeded. This paper presents the Merge, Split and Cluster approach: each time a new DAG of operators is submitted, i) its operators are first merged with the already running operators, ii) if an operator’s load thus created exceeds the nodes’ capacity, the operators gets split into several instances, and iii) the operators of the resulting graph are clustered, each cluster being hosted by a single node so as to maximize intra-node traffic. Two heuristics are proposed for this last phase. Simulation results show that i) merging allows to drastically reduce the needs in computing resources, and ii) that the heuristic provides an efficient clustering minimizing the intra-node traffic. |
Year | DOI | Venue |
|---|---|---|
2020 | 10.1109/CCGrid49817.2020.00-86 | 2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID) |
Keywords | DocType | ISBN |
Stream Processing,Deployment,Clustering | Conference | 978-1-7281-6095-5 |
Citations | PageRank | References |
0 | 0.34 | 0 |
Authors | ||
2 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Aymen Jlassi | 1 | 0 | 1.01 |
| Cédric Tedeschi | 2 | 83 | 12.65 |