Paper Info
Title
Improving I/O Performance with Adaptive Data Compression for Big Data Applications
Abstract
Increasingly larger scale simulations are generating an unprecedented amount of data. However, the increasing gap between computation and I/O capacity on High End Computing machines makes a severe bottleneck for data analysis. As a solution, in-situ analytics processes output data while simulations are running and before placing data on disk. Data movement between simulation and analytics, however, incurs overheads of in-situ analytics at scale. This paper tries to answer the following question: can we use compression technology to reduce the data movement cost and improve the performance of in-situ analytics for peta-scale applications? In particular, we explore when, where, how to use the compression techniques to reduce data movement cost between simulation and analytics. To find out the best algorithm and place to compress data in given situation, we introduce an adaptive data compression algorithm in this paper. The adaptive compression service is developed and analyzed for the in-situ analytics middleware. Experimental results demonstrate that compression service increases data transition bandwidth and improve the application End-to-End transfer performance.
Year
DOI
Venue
2014
10.1109/IPDPSW.2014.138
IPDPS Workshops
Keywords
Field
DocType
i/o bottlenecks, in-situ analytics, compression, big data, high-end computing,bandwidth,big data,computational modeling,data transfer,data compression,compression,data analysis,data models,compression algorithms
Middleware,Data modeling,Data mining,Bottleneck,Data transmission,Computer science,Input/output,Analytics,Data compression,Big data,Computer engineering,Distributed computing
Conference
Citations 
PageRank 
References 
13
0.63
21
Authors
4
Name
Order
Citations
PageRank
Hongbo Zou1342.59
Yongen Yu2804.55
Wei Tang315210.65
Hsuan-Wei Michelle Chen41125.74