Name
Abstract | ||
|---|---|---|
Analysis of large-scale simulation output is a core element of scientific inquiry, but analysis queries may experience significant I/O overhead when the data is not structured for efficient retrieval. While in-situ processing allows for improved time-to-insight for many applications, scaling in-situ frameworks to hundreds of thousands of cores can be difficult in practice. The DeltaFS in-situ indexing is a new approach for in-situ processing of massive amounts of data to achieve efficient point and small-range queries. This paper describes the challenges and lessons learned when scaling this in-situ processing function to hundreds of thousands of cores. We propose techniques for scalable all-to-all communication that is memory and bandwidth efficient, concurrent indexing, and specialized LSM-Tree formats. Combining these techniques allows DeltaFS to control the cost of in-situ processing while maintaining 3 orders of magnitude query speedup when scaling alongside the popular VPIC particle-in-cell code to 131,072 cores. |
Year | DOI | Venue |
|---|---|---|
2018 | 10.1109/SC.2018.00006 | SC |
Keywords | Field | DocType |
Indexing,Data models,Computational modeling,Trajectory,Analytical models,Libraries | Bandwidth efficient,Data modeling,Orders of magnitude (numbers),Computer science,Parallel computing,Search engine indexing,Scaling,Trajectory,Scalability,Speedup | Conference |
ISBN | Citations | PageRank |
978-1-5386-8384-2 | 4 | 0.44 |
References | Authors | |
42 | 9 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Qing Zheng | 1 | 91 | 5.40 |
| Charles D. Cranor | 2 | 582 | 52.19 |
| Danhao Guo | 3 | 4 | 0.44 |
| Gregory R. Ganger | 4 | 4560 | 383.16 |
| George Amvrosiadis | 5 | 111 | 10.40 |
| Garth A. Gibson | 6 | 849 | 61.69 |
| Bradley W. Settlemyer | 7 | 120 | 13.00 |
| Gary Grider | 8 | 253 | 16.11 |
| Fan Guo | 9 | 438 | 18.96 |