Efficient Query Evaluation Using A Two-level Retrieval Process

Abstract

We present an efficient query evaluation method based on a two level approach: at the first level, our method iterates in parallel over query term postings and identifies candidate documents using an <i>approximate evaluation</i> taking into account only partial information on term occurrences and no query independent factors; at the second level, promising candidates are <i>fully evaluated</i> and their exact scores are computed. The efficiency of the evaluation process can be improved significantly using dynamic pruning techniques with very little cost in effectiveness. The amount of pruning can be controlled by the user as a function of time allocated for query evaluation. Experimentally, using the TREC Web Track data, we have determined that our algorithm significantly reduces the total number of full evaluations by more than 90%, almost <i>without any loss</i> in precision or recall. At the heart of our approach there is an efficient implementation of a new Boolean construct called <b>WAND</b> or <i>Weak AND</i> that might be of independent interest.