QLoRA: Efficient Finetuning of Quantized LLMs

Abstract

We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLoRA backpropagates gradients through the frozen 4-bit quantization of a pretrained language model into low-rank adapters~(LoRA). We name our best model family Guanaco, showing that it outperforms all previously openly released models on the Vicuna benchmark, reaching 99.3% of the performance level of ChatGPT while only requiring 24 hours of finetuning on a single GPU. QLoRA introduces a number of innovations to save memory without sacrificing performance: (a) a new data type, 4-bit NormalFloat (NF4), which is information-theoretically optimal for normally distributed weights (b) double quantization to reduce the average memory footprint, and (c) Paged Optimizers to manage memory spikes from gradient checkpointing. We use QLoRA to finetune more than 1,000 models, providing a detailed analysis of instruction following performance across 9 instruction datasets, model types (LLaMA, T5), including on model scales that would be infeasible to run with regular finetuning (e.g. 30B and 65B parameter models). Our results show that QLoRA finetuning on a small high-quality dataset leads to state-of-the-art results, even when using smaller models than the previous SoTA. We release all of our models and code, including CUDA kernels for 4-bit training and inference.