Name
Abstract | ||
|---|---|---|
Many science facilities, such as the Department of Energy's Leadership Computing Facilities and experimental facilities including the Spallation Neutron Source, Stanford Linear Accelerator Center, and Advanced Photon Source, produce massive amounts of experimental and simulation data. These data are often shared among the facilities and with collaborating institutions. Moving large datasets over the wide-area network (WAN) is a major problem inhibiting collaboration. Next-generation, terabit-networks will help alleviate the problem, however, the parallel storage systems on the endsystem hosts at these institutions can become a bottleneck for terabit data movement. The parallel storage system (PFS) is shared by simulation systems, experimental systems, analysis and visualization clusters, in addition to wide-area data movers. These competing uses often induce temporary, but significant, I/O load imbalances on the storage system, which impact the performance of all the users. The problem is a serious concern because some resources are more expensive (e.g. super computers) or have time-critical deadlines (e.g. experimental data from a light source), but parallel file systems handle all requests fairly even if some storage servers are under heavy load. This paper investigates the problem of competing workloads accessing the parallel file system and how the performance of wide-area data movement can be improved in these environments. First, we study the I/O load imbalance problems using actual I/O performance data collected from the Spider storage system at the Oak Ridge Leadership Computing Facility. Second, we present I/O optimization solutions with layout-awareness on end-system hosts for bulk data movement. With our evaluation, we show that our I/O optimization techniques can avoid the I/O congested disk groups, improving storage I/O times on parallel storage systems for terabit data movement. |
Year | DOI | Venue |
|---|---|---|
2013 | 10.1109/BigData.2013.6691661 | BigData Conference |
Keywords | Field | DocType |
department of energy,wide-area data movers,io performance data,io load imbalances,scheduling,storage io times,visualization clusters,terabit data movement,next-generation terabit-networks,end-system hosts,wide-area network,spider storage system,advanced photon source,natural sciences computing,resource allocation,time-critical deadlines,leadership computing facilities,pfs,input-output programs,spallation neutron source,oak ridge leadership computing facility,experimental facilities,layout-aware io scheduling,science facilities,wan,networking,parallel storage systems,parallel file systems,i/o scheduling,storage systems,wide area networks,io optimization solutions,bulk data movement,terabits data movement,io congested disk groups,layout-awareness,parallel databases,stanford linear accelerator center | Data mining,Bottleneck,File system,I/O scheduling,Computer science,Scheduling (computing),Computer data storage,Server,Resource allocation,Terabit,Operating system,Distributed computing | Conference |
ISSN | Citations | PageRank |
2639-1589 | 3 | 0.50 |
References | Authors | |
6 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Youngjae Kim | 1 | 590 | 61.25 |
| Scott Atchley | 2 | 186 | 15.09 |
| Geoffroy Vallée | 3 | 123 | 15.62 |
| Galen M. Shipman | 4 | 299 | 22.83 |