Reformer: The Efficient Transformer

Abstract

Large Transformer models routinely achieve state-of-the-art results ona number of tasks but training these models can be prohibitively costly,especially on long sequences. We introduce two techniques to improvethe efficiency of Transformers. For one, we replace dot-product attentionby one that uses locality-sensitive hashing, changing its complexityfrom O(L^2) to O(L), where L is the length of the sequence.Furthermore, we use reversible residual layers instead of the standardresiduals, which allows storing activations only once in the trainingprocess instead of N times, where N is the number of layers.The resulting model, the Reformer, performs on par with Transformer modelswhile being much more memory-efficient and much faster on long sequences.