Name
Abstract | ||
|---|---|---|
Current peta-scale data analytics frameworks suffer from a significant performance bottleneck due to an imbalance between their enormous computational power and limited I/O bandwidth. Using data compression schemes to reduce the amount of I/O activity is a promising approach to addressing this problem. In this paper, we propose a hybrid framework for interleaving I/O with data compression to achieve improved I/O throughput side-by-side with reduced dataset size. We evaluate several interleaving strategies, present theoretical models, and evaluate the efficiency and scalability of our approach through comparative analysis. With our theoretical model, considering 19 real-world scientific datasets both from the public domain and peta-scale simulations, we estimate that the hybrid method can result in a 12 to 46 increase in throughput on hard-to-compress scientific datasets. At the reported peak bandwidth of 60 GB/s of uncompressed data for a current, leadership-class parallel I/O system, this translates into an effective gain of 7 to 28 GB/s in aggregate throughput. |
Year | DOI | Venue |
|---|---|---|
2012 | 10.1145/2287076.2287086 | HPDC |
Keywords | Field | DocType |
isobar hybrid compression-i,o interleaving,o system,data compression scheme,hybrid framework,o optimization,hard-to-compress scientific datasets,large-scale parallel,o activity,aggregate throughput,o bandwidth,uncompressed data,current peta-scale data,o throughput,i o,data compression,public domain,staging,comparative analysis,high performance computing,lossless compression,isobar | Data analysis,Computer science,Parallel computing,Real-time computing,Input/output,Throughput,Parallel I/O,Data compression,Interleaving,Lossless compression,Scalability | Conference |
Citations | PageRank | References |
19 | 0.89 | 27 |
Authors | ||
12 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Eric R. Schendel | 1 | 61 | 5.02 |
| Saurabh V. Pendse | 2 | 48 | 3.33 |
| John Jenkins | 3 | 19 | 0.89 |
| David A. Boyuka II | 4 | 82 | 5.52 |
| Zhenhuan Gong | 5 | 351 | 13.71 |
| Sriram Lakshminarasimhan | 6 | 187 | 10.01 |
| Qing Liu | 7 | 389 | 25.62 |
| Hemanth Kolla | 8 | 250 | 17.13 |
| Jackie Chen | 9 | 80 | 4.62 |
| Scott Klasky | 10 | 1547 | 99.00 |
| Robert Ross | 11 | 2717 | 173.13 |
| Nagiza F. Samatova | 12 | 861 | 74.04 |