Title | ||
---|---|---|
Software-defined storage for fast trajectory queries using a deltaFS indexed massive directory. |
Abstract | ||
---|---|---|
In this paper we introduce the Indexed Massive Directory, a new technique for indexing data within DeltaFS. With its design as a scalable, server-less file system for HPC platforms, DeltaFS scales file system metadata performance with application scale. The Indexed Massive Directory is a novel extension to the DeltaFS data plane, enabling in-situ indexing of massive amounts of data written to a single directory simultaneously, and in an arbitrarily large number of files. We achieve this through a memory-efficient indexing mechanism for reordering and indexing data, and a log-structured storage layout to pack small writes into large log objects, all while ensuring compute node resources are used frugally. We demonstrate the efficiency of this indexing mechanism through VPIC, a widely-used simulation code that scales to trillions of particles. With DeltaFS, we modify VPIC to create a file for each particle to receive writes of that particle's output data. Dynamically indexing the directory's underlying storage allows us to achieve a 5000x speedup in single particle trajectory queries, which require reading all data for a single particle. This speedup increases with application scale while the overhead is fixed at 3% of available memory.
|
Year | DOI | Venue |
---|---|---|
2017 | 10.1145/3149393.3149398 | SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis
Denver
Colorado
November, 2017 |
DocType | ISBN | Citations |
Conference | 978-1-4503-5134-8 | 1 |
PageRank | References | Authors |
0.35 | 0 | 8 |
Name | Order | Citations | PageRank |
---|---|---|---|
Qing Zheng | 1 | 91 | 5.40 |
George Amvrosiadis | 2 | 111 | 10.40 |
Saurabh Kadekodi | 3 | 2 | 1.71 |
Garth A. Gibson | 4 | 849 | 61.69 |
Charles D. Cranor | 5 | 582 | 52.19 |
Bradley W. Settlemyer | 6 | 120 | 13.00 |
Gary Grider | 7 | 253 | 16.11 |
Fan Guo | 8 | 12 | 5.25 |