Ranking Robustness: A Novel Framework To Predict Query Performance

Abstract

In this paper, we introduce the notion of ranking robustness, which refers to a property of a ranked list of documents that indicates how stable the ranking is in the presence of uncertainty in the ranked documents. We propose a statistical measure called the robustness score to quantify this notion. We demonstrate that the robustness score significantly and consistently correlates with query performance in a variety of TREC test collections including the GOV2 collection. We compare the robustness score with the clarity score method which is the state-of-the-art technique for query performance prediction. Our experimental results show that the robustness score performs better than or at least as good as the clarity score. We find that the clarity score is barely correlated with query performance on the GOV2 collection while the correlation between the robustness score and query performance remains significant. We also notice that a combination of the two usually results in more prediction power.