Name
Abstract | ||
|---|---|---|
Detection of near duplicate documents is an important problem in many data mining and information filtering applications. When faced with massive quantities of data, traditional techniques relying on direct inter-document similarity computation are often not feasible given the time and memory performance constraints. On the other hand, fingerprint-based methods, such as I-Match, while very attractive computationally, can be unstable even to small perturbations of document content, which causes signature fragmentation. We focus on I-Match and present a randomization-based technique of increasing its signature stability, with the proposed method consistently outperforming traditional I-Match by as high as 40---60% in terms of the relative improvement in near-duplicate recall. Importantly, the large gains in detection accuracy are offset by only small increases in computational requirements. We also address the complimentary problem of spurious matches, which is particularly important for I-Match when fingerprinting long documents. Our discussion is supported by experiments involving large web-page and email datasets. |
Year | DOI | Venue |
|---|---|---|
2008 | 10.1007/s11227-007-0171-z | The Journal of Supercomputing |
Keywords | Field | DocType |
Information retrieval efficiency,Spam detection | Similitude,Data mining,Supercomputer,Computer science,Filter (signal processing),Fingerprint,Information extraction,Spurious relationship,Offset (computer science),Data link | Journal |
Volume | Issue | ISSN |
45 | 3 | 0920-8542 |
Citations | PageRank | References |
8 | 0.57 | 25 |
Authors | ||
2 | ||
Name | Order | Citations | PageRank |
|---|---|---|---|
| Aleksander Kołcz | 1 | 628 | 66.65 |
| Abdur Chowdhury | 2 | 2013 | 160.59 |