Paper Info
Title
HBA: Distributed Metadata Management for Large Cluster-Based Storage Systems
Abstract
An efficient and distributed scheme for file mapping or file lookup is critical in decentralizing metadata management within a group of metadata servers. This paper presents a novel technique called HBA (Hierarchical Bloom filter Arrays) to map filenames to the metadata servers holding their metadata. Two levels of probabilistic arrays, namely, Bloom filter arrays, with different level of accuracies, are used on each metadata server. One array, with lower accuracy and representing the distribution of the entire metadata, trades accuracy for significantly reduced memory overhead, while the other array, with higher accuracy, caches partial distribution information and exploits the temporal locality of file access patterns. Both arrays are replicated to all metadata servers to support fast local lookups. We evaluate HBA through extensive trace-driven simulations and an implementation in Linux. Simulation results show our HBA design to be highly effective and efficient in improving performance and scalability of file systems in clusters with 1,000 to 10,000 nodes (or super-clusters) and with the amount of data in the Peta-byte scale or higher. Our implementation indicates that HBA can reduce metadata operation time of a single-metadata-server architecture by a factor of up to 43.9 when the system is configured with 16 metadata servers.
Year
DOI
Venue
2008
10.1109/TPDS.2007.70788
IEEE Trans. Parallel Distrib. Syst.
Keywords
DocType
Volume
entire metadata,file system,metadata management,large cluster-based storage systems,metadata operation time,file access pattern,file mapping,hba design,metadata server,lower accuracy,higher accuracy,distributed computing,file servers,distributed system,computer networks,meta data,linux,scalability,distributed systems,bandwidth,storage system,parallel systems,bloom filter,distributed file system
Journal
19
Issue
ISSN
Citations 
6
1045-9219
40
PageRank 
References 
Authors
1.55
31
4
Name
Order
Citations
PageRank
Yifeng Zhu151335.33
Hong Jiang2431.93
Jun Wang342025.60
Feng Xian4592.68