Name
Abstract | ||
|---|---|---|
The processing of massive small files is a challenge in the design of distributed file systems. Currently, the combined-block-storage approach is prevalent. However, the approach employs traditional file systems like ExtFS and may cause inefficiency for random access to small files. This paper focuses on optimizing the performance of data servers in accessing massive small files. We present a Flat Lightweight File System (iFlatLFS) to manage small files, which is based on a simple metadata scheme and a flat storage architecture. iFlatLFS aims to substitute the traditional file system on data servers that are mainly used to store small files, and it can greatly simplify the original data access procedure. The new metadata proposed in this paper occupies only a fraction of the original metadata size based on traditional file systems. We have implemented iFlatLFS in CentOS 5.5 and integrated it into an open source Distributed File System (DFS), called Taobao FileSystem (TFS), which is developed by a top B2C service provider, Alibaba, in China and is managing over 28.6 billion small photos. We have conducted extensive experiments to verify the performance of iFlatLFS. The results show that when the file size ranges from 1KB to 64KB, iFlatLFS is faster than Ext4 by 48% and 54% on average for random read and write in the DFS environment, respectively. Moreover, after iFlatLFS is integrated into TFS, iFlatLFS-based TFS is faster than the existing Ext4-based TFS by 45% and 49% on average for random read access and hybrid access (the mix of read and write accesses), respectively. |
Year | DOI | Venue |
|---|---|---|
2013 | 10.1109/HiPC.2013.6799116 | HiPC |
Keywords | Field | DocType |
iflatlfs,dfs,storage management,data access procedure simplification,information retrieval,data structures,flat storage architecture,data server performance optimization,massive small file processing,combined-block-storage approach,flat lightweight file system,metadata scheme,meta data,open source distributed file system,distributed databases | File Control Block,Virtual file system,File system,Computer science,Indexed file,File system fragmentation,Distributed computing,Parallel computing,File size,Torrent file,Operating system,Database,Computer file | Conference |
ISSN | Citations | PageRank |
1094-7256 | 0 | 0.34 |
References | Authors | |
8 | 8 | |
Name | Order | Citations | PageRank |
|---|---|---|---|
| Songling Fu | 1 | 12 | 2.89 |
| Chenlin Huang | 2 | 48 | 8.83 |
| Ligang He | 3 | 542 | 56.73 |
| Nadeem Chaudhary | 4 | 6 | 2.15 |
| Xiangke Liao | 5 | 622 | 74.79 |
| Shazhou Yang | 6 | 0 | 0.34 |
| Xiaochuan Wang | 7 | 0 | 0.68 |
| Bao Li | 8 | 0 | 0.34 |