Paper Info
Title
Performance Optimization for Managing Massive Numbers of Small Files in Distributed File Systems
Abstract
The processing of massive numbers of small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs the traditional file systems such as ExtFS and may cause inefficiency when accessing small files randomly located in the disk. This paper focuses on optimizing the performance of data servers in accessing massive numbers of small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS is designed to substitute the traditional file system on data servers and can be deployed underneath distributed file systems that store massive numbers of small files. iFlatLFS can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1KB to 64KB, iFlatLFS is faster than Ext4 by 48% and 54% on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45% and 49% on average for random read access and hybrid access (the mix of read and write accesses), respectively.
Year
DOI
Venue
2015
10.1109/TPDS.2014.2377720
IEEE Transactions on Parallel and Distributed Systems
Keywords
Field
DocType
data server,distributed file system,performance optimization,small file,optimization,distributed databases,servers,data models
File Control Block,File system,Computer science,Torrent file,File size,Unix file types,Versioning file system,File system fragmentation,Operating system,Database,Distributed computing,Computer file
Journal
Volume
Issue
ISSN
PP
99
1045-9219
Citations 
PageRank 
References 
6
0.45
19
Authors
5
Name
Order
Citations
PageRank
Songling Fu1122.89
Ligang He254256.73
Chenlin Huang3488.83
Xiangke Liao462274.79
Kenli Li51389124.28