Paper Info
Title
Key/Value-Enabled Flash Memory for Complex Scientific Workflows with On-Line Analysis and Visualization
Abstract
Scientific workflows are often composed of compute-intensive simulations and data-intensive analysis and visualization, both equally important for productivity. High-performance computers run the compute-intensive phases efficiently, but data-intensive processing is still getting less attention. Dense non-volatile memory integrated into super-computers can help address this problem. In addition to density, it offers significantly finer-grained I/O than disk-based I/O systems. We present a way to exploit the fundamental capabilities of Storage-Class Memories (SCM), such as Flash, by using scalable key-value (KV) I/O methods instead of traditional file I/O calls commonly used in HPC systems. Our objective is to enable higher performance for on-line and near-line storage for analysis and visualization of very high resolution, but correspondingly transient, simulation results. In this paper, we describe 1) the adaptation of a scalable key-value store to a BlueGene/Q system with integrated Flash memory, 2) a novel key-value aggregation module which implements coalesced, function-shipped calls between the clients and the servers, and 3) the refactoring of a scientific workflow to use application-relevant keys for fine-grained data subsets. The resulting implementation is analogous to function-shipping of POSIX I/O calls but shows an order of magnitude increase in read and a factor 2.5x increase in write IOPS performance (11 million read IOPS, 2.5 million write IOPS from 4096 compute nodes) when compared to a classical file system on the same system. It represents an innovative approach for the integration of SCM within an HPC system at scale.
Year
DOI
Venue
2016
10.1109/IPDPS.2016.23
2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS)
Keywords
Field
DocType
function shipping,key-value store,SCM,Flash,online analysis,visualization
Data visualization,File system,Flash memory,Visualization,Computer science,IOPS,Server,Parallel computing,POSIX,Operating system,Distributed computing,Scalability
Conference
ISSN
ISBN
Citations 
1530-2075
978-1-5090-2141-3
4
PageRank 
References 
Authors
0.40
12
16
Name
Order
Citations
PageRank
Stefan Eilemann11178.92
Fabien Delalondre2303.74
Jon Bernard340.40
Judit Planas443822.43
Felix Schürmann524527.04
John Biddiscombe6728.42
Costas Bekas78112.75
Alessandro Curioni827939.87
Bernard Metzler9897.72
Peter Kaltstein1040.40
Peter Morjan11120.96
Joachim Fenkes12121.30
Ralph Bellofatto1329035.67
Lars Schneidenbach14497.02
T. J. Christopher Ward1525234.78
Blake G. Fitch1625232.82