Paper Info
Title
A Logistic Based Mathematical Model to Optimize Duplicate Elimination Ratio in Content Defined Chunking Based Big Data Storage System.
Abstract
Deduplication is an efficient data reduction technique, and it is used to mitigate the problem of huge data volume in big data storage systems. Content defined chunking (CDC) is the most widely used algorithm in deduplication systems. The expected chunk size is an important parameter of CDC, and it influences the duplicate elimination ratio (DER) significantly. We collected two realistic datasets to perform an experiment. The experimental results showed that the current approach of setting the expected chunk size to 4 KB or 8 KB empirically cannot optimize DER. Therefore, we present a logistic based mathematical model to reveal the hidden relationship between the expected chunk size and the DER. This model provides a theoretical basis for optimizing DER by setting the expected chunk size reasonably. We used the collected datasets to verify this model. The experimental results showed that the R-2 values, which describe the goodness of fit, are above 0.9, validating the correctness of this mathematic model. Based on the DER model, we discussed how to make DER close to the optimum by setting the expected chunk size reasonably.
Year
DOI
Venue
2016
10.3390/sym8070069
SYMMETRY-BASEL
Keywords
Field
DocType
storage system,deduplication,duplication elimination ratio,content defined chunking
Data deduplication,Data mining,Computer data storage,Computer science,Correctness,Chunking (psychology),Big data,Goodness of fit,Data reduction
Journal
Volume
Issue
Citations 
8
7
2
PageRank 
References 
Authors
0.39
11
6
Name
Order
Citations
PageRank
Longxiang Wang182.01
Xiaoshe Dong217251.44
Xingjun Zhang38134.06
Fuliang Guo440.89
Yinfeng Wang56113.10
Weifeng Gong682.51