Document Clustering Using Word Clusters Via The Information Bottleneck Method

Abstract

We present a novel implementation of the recently introduced <i>information bottleneck method</i> for unsupervised document clustering. Given a joint empirical distribution of words and documents, <i>p</i>(<i>x</i>, <i>y</i>), we first cluster the words, <i>Y</i>, so that the obtained word clusters, Ytilde;, maximally preserve the information on the documents. The resulting joint distribution. <i>p</i>(<i>X</i>, <i>Ytilde;</i>), contains most of the original information about the documents, <i>I</i>(<i>X</i>; <i>Ytilde;</i>) ≈ <i>I</i>(<i>X</i>; <i>Y</i>), but it is much less sparse and noisy. Using the same procedure we then cluster the documents, <i>X</i>, so that the information about the word-clusters is preserved. Thus, we first find <i>word-clusters</i> that capture most of the mutual information about to set of documents, and then find <i>document clusters</i>, that preserve the information about the word clusters. We tested this procedure over several document collections based on subsets taken from the standard 20<i>Newsgroups</i> corpus. The results were assessed by calculating the correlation between the document clusters and the correct labels for these documents. Finding from our experiments show that this <i>double clustering</i> procedure, which uses the information bottleneck method, yields significantly superior performance compared to other common document distributional clustering algorithms. Moreover, the double clustering procedure improves all the distributional clustering methods examined here.