Paper Info
Title
ShardFS vs. IndexFS: replication vs. caching strategies for distributed metadata management in cloud storage systems
Abstract
The rapid growth of cloud storage systems calls for fast and scalable namespace processing. While few commercial file systems offer anything better than federating individually non-scalable namespace servers, a recent academic file system, IndexFS, demonstrates scalable namespace processing based on client caching of directory entries and permissions (directory lookup state) with no per-client state in servers. In this paper we explore explicit replication of directory lookup state in all servers as an alternative to caching this information in all clients. Both eliminate most repeated RPCs to different servers in order to resolve hierarchical permission tests. Our realization for server replicated directory lookup state, ShardFS, employs a novel file system specific hybrid optimistic and pessimistic concurrency control favoring single object transactions over distributed transactions. Our experimentation suggests that if directory lookup state mutation is a fixed fraction of operations (strong scaling for metadata), server replication does not scale as well as client caching, but if directory lookup state mutation is proportional to the number of jobs, not the number of processes per job, (weak scaling for metadata), then server replication can scale more linearly than client caching and provide lower 70 percentile response times as well.
Year
DOI
Venue
2015
10.1145/2806777.2806844
IEEE International System-on-Chip (SoC) Conference
Keywords
Field
DocType
metadata management,caching,replication
Metadata,File system,Concurrency control,Computer science,Directory,Server,Computer network,Namespace,Distributed transaction,Cloud storage,Operating system
Conference
Citations 
PageRank 
References 
14
0.60
32
Authors
4
Name
Order
Citations
PageRank
Lin Xiao11024.97
Kai Ren222912.85
Qing Zheng3915.40
Garth A. Gibson42517250.27