Paper Info
Title
High performance RDMA-based design of HDFS over InfiniBand
Abstract
Hadoop Distributed File System (HDFS) acts as the primary storage of Hadoop and has been adopted by reputed organizations (Facebook, Yahoo! etc.) due to its portability and fault-tolerance. The existing implementation of HDFS uses Java-socket interface for communication which delivers suboptimal performance in terms of latency and throughput. For data-intensive applications, network performance becomes key component as the amount of data being stored and replicated to HDFS increases. In this paper, we present a novel design of HDFS using Remote Direct Memory Access (RDMA) over InfiniBand via JNI interfaces. Experimental results show that, for 5GB HDFS file writes, the new design reduces the communication time by 87% and 30% over 1Gigabit Ethernet (1GigE) and IP-over-InfiniBand (IPoIB), respectively, on QDR platform (32Gbps). For HBase, the Put operation performance is improved by 26% with our design. To the best of our knowledge, this is the first design of HDFS over InfiniBand networks.
Year
DOI
Venue
2012
10.1109/SC.2012.65
SC
Keywords
Field
DocType
communication time,rdma-based design,operation performance,high performance,suboptimal performance,network performance,hdfs file,file system,novel design,hdfs increase,infiniband network,new design,distributed databases,local area networks,public domain software,gpu programming
Distributed File System,InfiniBand,Computer science,Parallel computing,Ethernet,Remote direct memory access,Software portability,Local area network,Throughput,Operating system,Network performance,Distributed computing
Conference
ISSN
ISBN
Citations 
2167-4329
978-1-4673-0804-5
34
PageRank 
References 
Authors
1.52
17
8
Name
Order
Citations
PageRank
Nazrul Islam19612.24
Md. Mahmudur Rahman265250.91
Jithin Jose340625.30
Raghunath Rajachandrasekar416010.52
H. Wang58415.66
H. Subramoni61026.79
C. Siva Ram Murthy72020189.72
Dhabaleswar K. Panda85366446.70