Name
Abstract | ||
|---|---|---|
Persistent data structures represent a core component of high-performance data analytics. Multiple data processing systems persist data structures using memory-mapped files. Memory-mapped file I/O provides a productive and unified programming interface to different types of storage systems. However, it suffers from multiple limitations, including performance bottlenecks caused by system-wide configurations and a lack of support for efficient incremental versioning. There-fore, many such systems only support versioning via full-copy snapshots, resulting in poor performance and storage capacity bottlenecks. To address these limitations, we present Privateer 2.0, a virtual memory and storage interface that optimizes performance and storage capacity for versioned persistent data structures. Privateer 2.0 improves over the previous version by supporting userspace virtual memory management and block compression. We integrated Privateer 2.0 into Metall, a C++ persistent data structure allocator, and LMDB, a widely-used key-value store database. Privateer 2.0 yielded up to 7.5× speedup and up to 300× storage space reduction for Metall incremental snapshots and 1.25× speedup with 11.7× storage space reduction for LMDB incremental snapshots. |
Year | DOI | Venue |
|---|---|---|
2022 | 10.1109/HPEC55821.2022.9926392 | 2022 IEEE High Performance Extreme Computing Conference (HPEC) |
Keywords | DocType | ISSN |
high-performance data analytics,multiple data processing systems,memory-mapped files,productive programming interface,unified programming interface,storage systems,multiple limitations,performance bottlenecks,system-wide configurations,efficient incremental versioning,storage capacity bottlenecks,storage interface,versioned persistent data structures,userspace virtual memory management,block compression,integrated Privateer 2,persistent data structure allocator,storage space reduction,memory-mapped persistent data structures | Conference | 2377-6943 |
ISBN | Citations | PageRank |
978-1-6654-9787-9 | 0 | 0.34 |
References | Authors | |
11 | 5 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Karim Youssef | 1 | 0 | 0.34 |
| Abdullah Al Raqibul Islam | 2 | 0 | 0.34 |
| Keita Iwabuchi | 3 | 0 | 0.34 |
| Wu-chun Feng | 4 | 2812 | 232.50 |
| Roger Pearce | 5 | 0 | 0.34 |