Deeper Text Understanding For IR With Contextual Neural Language Modeling

Abstract

Identifying which terms are important with respect to a specific query or document is crucial to Information Retrieval, but context is difficult to capture with traditional term frequency signals. This paper proposes a Deep Contextualized Term Weighting framework (DeepCT) that identifies import terms by taking into consideration the meaning of the term and the role it plays in a specific context. The DeepCT framework is built upon the neural contextualized text representations of BERT[3]. It learns to map the contextualized word representations onto the target term weights in a supervised manner. In DeepCT, a specific term's weight may vary in different textual contexts, reflecting its relations to other words. The contextualized term weights from DeepCT are beneficial to both document and query understanding. On the document side, we propose DeepCT-Index, which uses DeepCT to estimate the importance of each document term, and then uses those estimations to generate term weights that are stored in the index. On the query side, we propose DeepCT-Query, which estimates the importance of each query term and use the weights to generate new query representations. Both the new index and the new query can be used directly by bag-of-words retrieval models such as BM25 and QL, making these methods efficient and easy to incorporate into existing search systems. Experiments demonstrate that DeepCT can greatly improve retrieval accuracy by providing a deeper understanding of a term's importance in a specific document/query context. expand