Name
Title | ||
|---|---|---|
Improving Collective Mpi-Io Using Topology-Aware Stepwise Data Aggregation With I/O Throttling |
Abstract | ||
|---|---|---|
MPI-IO has been used in an internal I/O interface layer of HDF5 or PnetCDF, where collective MPI-IO plays a big role in parallel I/O to manage a huge scale of scientific data. However, existing collective MPI-IO optimization named two-phase I/O has not been tuned enough for recent supercomputers consisting of mesh/torus interconnects and a huge scale of parallel file systems due to lack of topology-awareness in data transfers and optimization for parallel file systems. In this paper, we propose I/O throttling and topology-aware stepwise data aggregation in two-phase I/O of ROMIO, which is a representative MPI-IO library, in order to improve collective MPI-IO performance even if we have multiple processes per compute node. Throttling I/O requests going to a target file system mitigates I/O request contention, and consequently I/O performance improvements are achieved in file access phase of two-phase I/O. Topology-aware aggregator layout with paying attention to multiple aggregators per compute node alleviates contention in data aggregation phase of two-phase I/O. In addition, stepwise data aggregation improves data aggregation performance. HPIO benchmark results on the K computer indicate that the proposed optimization has achieved up to about 73% and 39% improvements in write performance compared with the original implementation using 12,288 and 24,576 processes on 3,072 and 6,144 compute nodes, respectively. |
Year | DOI | Venue |
|---|---|---|
2018 | 10.1145/3149457.3149464 | PROCEEDINGS OF THE INTERNATIONAL CONFERENCE ON HIGH PERFORMANCE COMPUTING IN ASIA-PACIFIC REGION (HPC ASIA 2018) |
Keywords | Field | DocType |
MPI-IO, two-phase I/O, aggregator, topology-aware stepwise data aggregation, I/O throttling | Hierarchical Data Format,File system,Interface layer,News aggregator,Computer science,Parallel computing,Torus,Input/output,Data aggregator,Bandwidth throttling | Conference |
Citations | PageRank | References |
1 | 0.36 | 14 |
Authors | ||
6 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Yuichi Tsujita | 1 | 62 | 12.62 |
| Atsushi Hori | 2 | 1 | 0.69 |
| Toyohisa Kameyama | 3 | 1 | 1.03 |
| Atsuya Uno | 4 | 87 | 12.94 |
| Fumiyoshi Shoji | 5 | 52 | 7.36 |
| Yutaka Ishikawa | 6 | 1449 | 188.06 |