Name
Abstract | ||
|---|---|---|
Two camps of file systems exist: parallel file systems designed for conventional high performance computing (HPC) and distributed file systems designed for newly emerged data-intensive applications. Addressing the big data challenge requires an approach that utilizes both high performance computing and data-intensive computing power. Thus, HPC applications may need to interact with distributed file systems, such as HDFS. The N-1 (N-to-1) parallel file write is a critical technical challenge, because it is very common for HPC applications but HDFS does not allow it. This study introduces a system solution, named SCALER, which allows MPI based applications to directly access HDFS without extra data movement. SCALER supports N-1 file write at both the inter-block level and intra-block level. Experimental results confirm that SCALER achieves the design goal efficiently. |
Year | DOI | Venue |
|---|---|---|
2014 | 10.1109/CLUSTER.2014.6968736 | CLUSTER |
Keywords | Field | DocType |
parallel processing,hdfs,optimization,distributed file systems,scaler,mpi based applications,parallel i/o,scalable parallel file write,data-intensive computing power,high performance computing,n-1 parallel file write,hpc,intrablock level,design goal,data-intensive applications,parallel file systems,message passing,interblock level,parallel databases,algorithm design and analysis,computational modeling,computer architecture,writing | Distributed File System,Self-certifying File System,Supercomputer,Computer science,Device file,Parallel computing,Parallel I/O,File system fragmentation,Operating system,Scalability,Computer file | Conference |
ISSN | Citations | PageRank |
1552-5244 | 1 | 0.36 |
References | Authors | |
0 | 4 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Xi Yang | 1 | 75 | 5.61 |
| Yanlong Yin | 2 | 134 | 8.93 |
| Hui Jin | 3 | 114 | 7.82 |
| Xian-he Sun | 4 | 1987 | 182.64 |