Finding Near-duplicate Web Pages: A Large-scale Evaluation Of Algorithms

Abstract

Broder et al.'s [3] shingling algorithm and Charikar's [4] random projection based approach are considered "state-of-the-art" algorithms for finding near-duplicate web pages. Both algorithms were either developed at or used by popular web search engines. We compare the two algorithms on a very large scale, namely on a set of 1.6B distinct web pages. The results show that neither of the algorithms works well for finding near-duplicate pairs <i>on the same site</i>, while both achieve high precision for near-duplicate pairs <i>on different sites</i>. Since Charikar's algorithm finds more near-duplicate pairs on <i>different</i> sites, it achieves a better precision overall, namely 0.50 versus 0.38 for Broder et al.'s algorithm. We present a combined algorithm which achieves precision 0.79 with 79% of the recall of the other algorithms.