Learned Cardinality Estimation: An In-depth Study

Abstract

Learned cardinality estimation (CE) has recently gained significant attention for replacing long-studied traditional CE with machine learning, especially for deep learning. However, these estimators were developed independently and have not been fairly or comprehensively compared in common settings. Most studies use a subset of IMDB data which is too simple to measure their limits and determine whether they are ready for real, complex data. Furthermore, they are regarded as black boxes, without a deep understanding of why large errors occur. In this paper, we first provide a taxonomy and a unified workflow of learned estimators for a better understanding of estimators. We next comprehensively compare recent learned CE methods that support joins, from a subset of tables to full IMDB and TPC-DS datasets. Under the experimental results, we then demystify the black-box models and analyze critical components that cause large errors. We also measure their impact on query optimization. Finally, based on the findings, we suggest realizable research opportunities. We believe that a deeper understanding of the behavior of existing methods can provide a more comprehensive and substantial framework for developing better estimators.