Paper Info
Title
Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload
Abstract
As the number of compute cores on modern parallel machines increases to more than hundreds of thousands, scalable and consistent I/O performance is becoming hard to obtain due to fluctuating file system performance. This fluctuation is often caused by rebuilding RAID disk from hardware failures or concurrent jobs competing for I/O. We present a mechanism that stripes across a dynamically-selected subset of I/O servers with the lightest workload to achieve the best I/O bandwidth available from the system. We implement this mechanism into an I/O software layer that enables memory-to-file data layout transformation and allows transparent file partitioning. File partitioning is a technique that divides data among a set of files and manages file access, making data appear as a single file to users. Experimental results on NERSC's Hopper indicate that our approach effectively isolates I/O variation on shared systems and improves overall I/O performance significantly.
Year
DOI
Venue
2013
10.1109/CLUSTER.2013.6702689
CLUSTER
Keywords
Field
DocType
collective i/o,parallel system,i/o software layer,raid,i/o performance,nersc's hopper,fluctuating i/o workload,i/o servers,transparent file partitioning,disc drives,input-output programs,dynamic file striping,hardware failures,file organisation,file access,compute cores,memory-to-file data layout transformation,fluctuating file system performance,raid disk,parallel netcdf,i/o variation,concurrent jobs,parallel machines,file partitioning
File system,I/O scheduling,Computer science,Device file,Parallel computing,Real-time computing,Input/output,Unix file types,Versioning file system,File system fragmentation,Operating system,Computer file
Conference
ISSN
Citations 
PageRank 
1552-5244
1
0.35
References 
Authors
22
5
Name
Order
Citations
PageRank
Seung Woo Son129631.43
Saba Sehrish2567.00
Wei-keng Liao3109587.98
Ron Oldfield440818.71
Alok N. Choudhary53441326.32