Paper Info
Title
Experiences of Converging Big Data Analytics Frameworks with High Performance Computing Systems.
Abstract
With the rapid development of big data analytics frameworks, many existing high performance computing (HPC) facilities are evolving new capabilities to support big data analytics workloads. However, due to the different workload characteristics and optimization objectives of system architectures, migrating data-intensive applications to HPC systems that are geared for traditional compute-intensive applications presents a new challenge. In this paper, we address a critical question on how to accelerate complex application that contains both data-intensive and compute-intensive workloads on the Tianhe-2 system by deploying an in-memory file system as data access middleware; we characterize the impact of storage architecture on data-intensive MapReduce workloads when using Lustre as the underlying file system. Based on our characterization and findings of the performance behaviors, we propose shared map output shuffle strategy and file metadata cache layer to alleviate the impact of metadata bottleneck. The evaluation of these optimization techniques shows up to 17% performance benefit for data-intensive workloads.
Year
DOI
Venue
2018
10.1007/978-3-319-69953-0_6
Lecture Notes in Computer Science
Keywords
DocType
Volume
High performance computing,Big data,Convergence,File system,Hadoop
Conference
10776
ISSN
Citations 
PageRank 
0302-9743
0
0.34
References 
Authors
0
4
Name
Order
Citations
PageRank
Peng Cheng100.34
Yutong Lu230753.61
Yunfei Du37214.62
Zhiguang Chen47918.83