Paper Info
Title
Understanding and improving computational science storage access through continuous characterization
Abstract
Computational science applications are driving a demand for increasingly powerful storage systems. While many techniques are available for capturing the I/O behavior of individual application trial runs and specific components of the storage system, continuous characterization of a production system remains a daunting challenge for systems with hundreds of thousands of compute cores and multiple petabytes of storage. As a result, these storage systems are often designed without a clear understanding of the diverse computational science workloads they will support.In this study, we outline a methodology for scalable, con tinuous, systemwide I/O characterization that combines storage device instrumentation, static file system analysis, and a new mechanism for capturing detailed application-level behavior. This methodology allows us to quantify systemwide trends such as the way application behavior changes with job size, the "burstiness" of the storage system, and the evolution of file system contents over time. The data also can be examined by application domain to determine the most prolific storage users and also investigate how their I/O strategies correlate with I/O performance. At the most detailed level, our characterization methodology can also be used to focus on individual applications and guide tuning efforts for those applications. We demonstrate the effectiveness of our methodology by performing a multilevel, two-month study of Intrepid, a 557 teraflop IBM Blue Gene/P system. During that time, we captured application-level I/O characterizations from 6,481 unique jobs spanning 38 science and engineering projects with up to 163,840 processes per job. We also captured patterns of I/O activity in over 8 petabytes of block device traffic and summarized the con tents of file systems containing over 191 million flies. We then used the results of our study to tune example applications, highlight trends that impact the design of future storage systems, and identify- - opportunities for improvement in I/O characterization methodology.
Year
DOI
Venue
2011
10.1109/MSST.2011.5937212
TOS
Keywords
DocType
Volume
o characterization methodology,computational science storage access,continuous characterization,p system,computational science application,powerful storage systems,storage management,i/o behavior,o behavior,o strategy,individual application trial run,production system,o characterization,future storage system,clear understanding,daunting challenge,improving computational science storage,storage system,static file system analysis,powerful storage system,storage device instrumentation,diverse computational science
Conference
7
Issue
ISSN
ISBN
3
2160-195X E-ISBN : 978-1-4577-0428-4
978-1-4577-0428-4
Citations 
PageRank 
References 
88
3.14
17
Authors
7
Name
Order
Citations
PageRank
Philip Carns11055.57
Kevin Harms219513.06
william allcock3883.14
charles w bacon4883.14
Samuel Lang525210.18
Robert Latham61348.57
Robert Ross72717173.13