An Explanation of In-context Learning As Implicit Bayesian Inference

Abstract

Large pretrained language models such as GPT-3 have the surprising ability to do in-context learning, where the model learns to do a downstream task simply by conditioning on a prompt consisting of input-output examples. Without being explicitly pretrained to do so, the language model learns from these examples during its forward pass without parameter updates on "out-of-distribution" prompts. In-context learning is hard to characterize because of the large scale of the pretraining data. In this paper, we study the role of the pretraining distribution on the emergence of in-context learning. We show that in-context learning emerges via Bayesian inference when the pretraining distribution is a mixture of HMMs. We theoretically characterize when in-context learning occurs despite the distribution mismatch between prompts and pretraining data. In contrast to messy large-scale pretraining datasets needed for in-context learning in natural language, we generate a family of small-scale synthetic datasets (GINC) where Transformer and LSTM language models both exhibit in-context learning.