Name
Title | ||
|---|---|---|
Error-controlled, progressive, and adaptable retrieval of scientific data with multilevel decomposition |
Abstract | ||
|---|---|---|
ABSTRACTExtreme-scale simulations and high-resolution instruments have been generating an increasing amount of data, which poses significant challenges to not only data storage during the run, but also post-processing where data will be repeatedly retrieved and analyzed for a long period of time. The challenges in satisfying a wide range of post-hoc analysis needs while minimizing the I/O overhead caused by inappropriate and/or excessive data retrieval should never be left unmanaged. In this paper, we propose a data refactoring, compressing, and retrieval framework capable of 1) fine-grained data refactoring with regard to precision; 2) incrementally retrieving and recomposing the data in terms of various error bounds; and 3) adaptively retrieving data in multi-precision and multi-resolution with respect to different analysis. With the progressive data re-composition and the adaptable retrieval algorithms, our framework significantly reduces the amount of data retrieved when multiple incremental precision are requested and/or the downstream analysis time when coarse resolution is used. Experiments show that the amount of data retrieved under the same progressively requested error bound using our framework is 64% less than that using state-of-the-art single-error-bounded approaches. Parallel experiments with up to 1, 024 cores and ~ 600 GB data in total show that our approach yields 1.36× and 2.52× performance over existing approaches in writing to and reading from persistent storage systems, respectively. |
Year | DOI | Venue |
|---|---|---|
2021 | 10.1145/3458817.3476179 | The International Conference for High Performance Computing, Networking, Storage, and Analysis |
Keywords | DocType | ISSN |
Data compression,error control,storage and I/O,data retrieval | Conference | 2167-4329 |
ISBN | Citations | PageRank |
978-1-6654-8390-2 | 0 | 0.34 |
References | Authors | |
24 | 10 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Xin Liang | 1 | 4 | 2.07 |
| Qian Gong | 2 | 0 | 1.01 |
| Jieyang Chen | 3 | 0 | 3.04 |
| Ben Whitney | 4 | 19 | 4.38 |
| Lipeng Wan | 5 | 5 | 3.79 |
| Qing Liu | 6 | 389 | 25.62 |
| David Pugmire | 7 | 0 | 1.35 |
| R.K. Archibald | 8 | 93 | 10.41 |
| Norbert Podhorszki | 9 | 2 | 2.73 |
| S. Klasky | 10 | 82 | 12.77 |