Minimal Probing: Supporting Expensive Predicates For Top-k Queries

Abstract

This paper addresses the problem of evaluating ranked <i>top-k</i> queries with expensive predicates. As major DBMSs now all support expensive user-defined predicates for Boolean queries, we believe such support for ranked queries will be even more important: First ranked queries often need to model user-specific concepts of preference, relevance, or similarity, which call for dynamic user-defined functions. Second, middleware systems must incorporate external predicates for integrating autonomous sources typically accessible only by per-object queries. Third, fuzzy joins are inherently expensive, as they are essentially user-defined operations that dynamically associate multiple relations. These predicates, being dynamically defined or externally accessed, cannot rely on index mechanisms to provide zero-time sorted output, and must instead require per-object probe to evaluate. The current standard sort-merge framework for ranked queries cannot efficiently handle such predicates because it must completely probe all objects, before sorting and merging them to produce <i>top-k</i> answers. To minimize expensive probes, we thus develop the formal principle of "necessary probes," which determines if a probe is absolutely required. We then propose Algorithm <i>MPro</i> which, by implementing the principle, is provably optimal with minimal probe cost. Further, we show that <i>MPro</i> can scale well and can be easily parallelized. Our experiments using both a real-estate benchmark database and synthetic datasets show that <i>MPro</i> enables significant probe reduction, which can be orders of magnitude faster than the standard scheme using complete probing.