LoRA & QLoRA

Introduction

In the era of foundation models, fine-tuning remains the primary mechanism for adapting a general-purpose language model to a specific task, domain, or style. But fine-tuning a model with billions of parameters is brutally expensive. Every parameter needs a gradient, every gradient needs an optimizer state, and all of that must fit in GPU memory simultaneously.

For a 7B parameter model stored in float16, the weights alone occupy 14 GB. Full fine-tuning with AdamW requires storing the weights, gradients, and two optimizer states (first and second moment estimates), totaling roughly 4x the model size in memory — around 56 GB before you even account for activations. A 70B model? That demands over 500 GB, which exceeds even the largest single-GPU setups available today.

Low-Rank Adaptation (LoRA), introduced by Hu et al. in 2021, dramatically changes this equation. Instead of updating all parameters, LoRA freezes the pretrained weights and injects small, trainable low-rank matrices alongside them. The result: you train fewer than 1% of the original parameters while matching or approaching full fine-tuning quality. QLoRA, from Dettmers et al. in 2023, extends this by quantizing the frozen base model to 4 bits, enabling fine-tuning of a 65B model on a single 48 GB GPU.

The Fine-tuning Problem

Parameter counts at scale

A transformer model's parameters are distributed across its layers. For a model with L layers, hidden dimension d, and intermediate FFN dimension d_ff (typically 4d), each layer contains:

• Self-attention projections: W_Q, W_K, W_V, W_O — each d × d, totaling 4d² parameters
• Feed-forward network: Two matrices of d × d_ff and d_ff × d — totaling 8d² parameters (when d_ff = 4d)
• Layer norms: 4d parameters (negligible at scale)

That gives roughly 12d² parameters per layer. For LLaMA-7B with d = 4096 and L = 32, that is 32 × 12 × 4096² ≈ 6.4 billion parameters in the transformer blocks alone, plus the embedding and head layers.

The memory bottleneck

Full fine-tuning with mixed-precision AdamW requires storing, for every trainable parameter:

• FP16 weights: 2 bytes per parameter
• FP16 gradients: 2 bytes per parameter
• FP32 optimizer states: first moment (4 bytes) + second moment (4 bytes) + FP32 master copy of weights (4 bytes) = 12 bytes per parameter

Total: 16 bytes per parameter. For a 7B model: 112 GB. For 70B: 1.12 TB. And that is before accounting for activation memory during the forward pass, which can easily double the requirement.

This memory wall motivated the search for parameter-efficient fine-tuning (PEFT) methods — techniques that adapt a model while training only a small fraction of its parameters. LoRA is the most widely adopted of these methods, and for good reason: it is simple, effective, and introduces no additional inference latency after merging.

Low-Rank Matrix Decomposition

Matrix rank and its implications

The rank of a matrix is the number of linearly independent rows (or equivalently, columns) it contains. A d × d matrix has maximum rank d, but many real-world matrices have effective rank far below their dimension. When a matrix has rank r << d, it can be exactly represented as the product of two smaller matrices:

This is the foundation of low-rank matrix factorization. Instead of storing d² parameters, we store 2dr parameters. When r = 8 and d = 4096, that is a compression ratio of 4096² / (2 × 4096 × 8) = 256x.

More practically, even when a matrix is technically full-rank, its information content is often concentrated in the top few singular values. The Eckart-Young-Mirsky theorem tells us that the best rank-r approximation of a matrix (in the Frobenius norm) is given by truncating its Singular Value Decomposition (SVD) to the top r singular values.

LoRA's key insight: weight updates are low-rank

The critical observation behind LoRA is not that pretrained weight matrices are low-rank — they generally are not. Instead, the change in weights during fine-tuning, ΔW = W_fine-tuned - W_pretrained, has very low intrinsic rank.

Hu et al. demonstrated this empirically by fine-tuning GPT-3 175B on various tasks and analyzing the resulting weight deltas. The singular value spectra of these deltas decay extremely rapidly, with the top few singular values capturing the overwhelming majority of the update's energy. Even rank r = 4 captures most of the adaptation signal for many tasks.

LoRA Mechanism

The modified forward pass

In standard fine-tuning, a pretrained weight matrix W₀ is updated to W₀ + ΔW. LoRA parameterizes this update as a low-rank product:

where B ∈ ℜ^d×r and A ∈ ℜ^r×d. The pretrained weights W₀ are frozen (no gradient computation) and only B and A are trained. At initialization, A is set to a random Gaussian matrix and B is initialized to zero, so ΔW = BA = 0 at the start of training — the model begins exactly where pretraining left off.

During the forward pass, the input x is projected through both the frozen W₀ and the LoRA branch simultaneously. The LoRA branch first projects x down to r dimensions (via A), then back up to d dimensions (via B). This bottleneck is what enforces the low-rank structure.

The rank parameter r

The rank r is LoRA's primary hyperparameter. It controls the expressiveness of the adaptation:

• r = 1: The update is a single outer product — a rank-1 perturbation. Extremely parameter-efficient but limited in capacity.
• r = 4–8: The most common setting. Sufficient for single-task fine-tuning on most downstream tasks. The original paper found r = 4 often matches full fine-tuning.
• r = 16–64: Used for more complex adaptations: multi-task learning, significant domain shifts, or instruction tuning across diverse tasks.
• r = 256+: Approaching full fine-tuning capacity. Rarely needed and loses most parameter efficiency benefits.

The number of trainable parameters per LoRA-adapted layer is 2 × d × r. For d = 4096 and r = 8, that is 65,536 parameters per adapted weight matrix — versus 16,777,216 parameters in the original matrix. A 256x reduction.

The alpha scaling factor

LoRA introduces a scaling factor α (alpha) that modulates the magnitude of the low-rank update:

The ratio α/r acts as a learned-rate multiplier for the LoRA branch. When α = r, the scaling factor is 1 and the LoRA update contributes at full strength. When α = 2r, the update is amplified by 2x.

The practical effect: α decouples the learning rate from the rank. If you double the rank r while keeping α fixed, each individual component of the low-rank factorization contributes half as much. This means you can sweep over ranks without retuning the learning rate each time. The convention in practice is to set α = 2r or α = r and tune the learning rate separately.

Which layers to adapt

LoRA can be applied to any linear layer, but not all layers benefit equally. The original paper focused on the attention weight matrices and found that adapting W_Q and W_V provided the best results for a given parameter budget. However, subsequent work and community practice have converged on a broader recipe:

• Attention projections (W_Q, W_K, W_V, W_O): The most impactful targets. These matrices control how the model attends to and combines information. Adapting all four is now standard practice.
• FFN layers (gate, up, down projections): Adapting these increases capacity significantly, especially for knowledge-intensive tasks or domain adaptation. Models like LLaMA use a gated FFN with three projections — adapting all of them is common in modern recipes.
• Embedding and LM head: Rarely adapted with LoRA. These layers have different structure (vocabulary dimension) and training dynamics. Some specialized tasks (new language tokens) may benefit.

The general rule: more target modules with a lower rank tends to outperform fewer modules with a higher rank, given the same total parameter budget. This distributes the adaptation signal more evenly across the network.

LoRA in Practice

Merge and deployment

One of LoRA's most compelling properties is zero-cost inference. After training, the low-rank matrices B and A can be merged back into the original weights:

This merged matrix has exactly the same shape as the original W₀, so inference uses the same architecture, the same memory footprint, and the same latency as the base model. No additional computation, no adapter overhead, no routing logic. The model is simply a standard transformer with different weight values.

This is a critical advantage over other PEFT methods. Prefix tuning adds extra tokens to the sequence (increasing attention cost). Adapter layers add new computation in the forward pass. LoRA, once merged, is invisible.

Multiple adapters and hot-swapping

Because LoRA adapters are small (typically 10–100 MB for a 7B model, versus 14 GB for the full weights), you can serve dozens of task-specific fine-tunes from a single base model. The workflow looks like this:

• At rest: Store one copy of the base model weights in GPU memory.
• Per request: Load the appropriate LoRA adapter (a few MB), apply it as an additive offset, run inference, then unload it.
• Batching: For concurrent requests with different adapters, frameworks like S-LoRA and Punica can batch requests sharing the same base model, applying per-request LoRA offsets efficiently using custom CUDA kernels.

This architecture enables multi-tenant serving: one GPU holds the base model, and each customer or task gets its own tiny adapter. You can even compose adapters by adding multiple ΔW matrices, though interference between adapters is an active area of research.

QLoRA: Quantized Low-Rank Adaptation

LoRA reduces the number of trainable parameters, but the frozen base model still must reside in GPU memory. For a 65B model in float16, that is 130 GB — still beyond a single high-end GPU. QLoRA (Dettmers et al., 2023) addresses this by quantizing the frozen base model to 4 bits while keeping the LoRA adapters in higher precision.

The key insight: since the base model weights are frozen during LoRA training, they do not need gradient-friendly precision. They only participate in the forward pass, where 4-bit precision introduces minimal error. The LoRA adapters, which do receive gradients, remain in BFloat16 to maintain training stability.

4-bit NormalFloat quantization

Standard integer quantization maps floating-point values to uniformly spaced integers. This works poorly for neural network weights because they follow an approximately normal distribution: most values cluster near zero, with rare outliers at the extremes.

NormalFloat4 (NF4) is an information-theoretically optimal 4-bit data type for normally distributed data. It works by:

1. Computing quantiles: Divide the standard normal distribution into 2⁴ = 16 regions of equal probability.
2. Assigning codewords: Each region maps to one 4-bit code. The quantization levels are not uniformly spaced — they are denser near zero (where most weights live) and sparser in the tails.
3. Normalizing: Before quantization, each weight block (typically 64 values) is normalized by its absolute maximum, so the distribution approximately matches a standard normal.

The NF4 quantization levels (for a standard normal) are approximately: -1.0, -0.6962, -0.5251, -0.3949, -0.2844, -0.1848, -0.0911, 0.0, 0.0796, 0.1609, 0.2461, 0.3379, 0.4407, 0.5626, 0.7230, 1.0. Notice the asymmetry and the denser spacing around zero.

Double quantization

Each block of 64 quantized weights requires a scaling factor (the absmax) stored in FP32 — that is 32 bits per 64 weights, or 0.5 extra bits per parameter. For a 65B model, those scaling factors alone consume over 4 GB.

QLoRA's double quantization quantizes the quantization constants themselves. The FP32 scaling factors are grouped into blocks of 256 and quantized to FP8, with their own (now much smaller) FP32 scaling factors. This reduces the overhead from 0.5 bits per parameter to approximately 0.127 bits per parameter — a saving of about 3 GB for a 65B model.

Paged optimizers

Even with 4-bit base weights and tiny LoRA adapters, optimizer states for the trainable parameters still need GPU memory. QLoRA introduces paged optimizers using NVIDIA's unified memory feature: when GPU memory is exhausted, optimizer state pages are automatically evicted to CPU RAM and paged back in on demand.

This is conceptually similar to virtual memory in operating systems. The GPU memory manager tracks which pages are "hot" (recently accessed) and which are "cold." During the backward pass, optimizer states for each layer are accessed sequentially, so the paging pattern is predictable and the overhead is modest — typically a 5–10% training slowdown in exchange for significantly reduced peak GPU memory.

In practice, paged optimizers are a safety net for memory spikes rather than a steady-state mechanism. With careful batch size and gradient accumulation settings, most QLoRA training runs stay within GPU memory. Paged optimizers prevent out-of-memory crashes during the occasional spike from long sequences or activation checkpointing boundary conditions.

Method Comparison

LoRA sits within a broader family of parameter-efficient fine-tuning methods. Each makes different tradeoffs between parameter efficiency, training cost, inference overhead, and implementation complexity.

LoRA's combination of high parameter efficiency, zero inference overhead (after merging), and strong empirical performance has made it the default PEFT method in the open-source LLM community. QLoRA extends this accessibility further, making fine-tuning of large models feasible on consumer hardware.

More recent developments build on LoRA's foundation. DoRA (Weight-Decomposed Low-Rank Adaptation) separates weight updates into magnitude and direction components. LoRA+ applies different learning rates to the A and B matrices. rsLoRA adjusts the scaling factor to maintain stable training at higher ranks. GaLore projects gradients into a low-rank subspace that rotates during training. All of these cite LoRA as their foundation.

Code Examples

LoRA fine-tuning with HuggingFace PEFT

from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from peft import LoraConfig, get_peft_model, TaskType
from trl import SFTTrainer

# ── Load base model ──────────────────────────────────────────
model_name = "meta-llama/Llama-2-7b-hf"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# ── Configure LoRA ───────────────────────────────────────────
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8,                        # rank — start here, increase if underfitting
    lora_alpha=16,              # alpha/r = 2 — moderate scaling
    lora_dropout=0.05,          # light dropout for regularization
    target_modules=[            # which layers to adapt
        "q_proj", "k_proj",     # attention queries and keys
        "v_proj", "o_proj",     # attention values and output
        "gate_proj",            # FFN gate (LLaMA gated FFN)
        "up_proj", "down_proj", # FFN up/down projections
    ],
    bias="none",                # don't train biases
)

# ── Wrap model with LoRA ─────────────────────────────────────
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622

# ── Training ─────────────────────────────────────────────────
training_args = TrainingArguments(
    output_dir="./lora-llama2-7b",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,             # higher LR than full FT (1e-5)
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    optim="adamw_torch",
)

trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=dataset,          # your formatted dataset
    tokenizer=tokenizer,
    max_seq_length=2048,
)
trainer.train()

QLoRA: 4-bit quantized fine-tuning

from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
import torch

# ── 4-bit quantization config (QLoRA) ───────────────────────
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",           # NormalFloat4 — optimal for normal distributions
    bnb_4bit_compute_dtype=torch.bfloat16,# compute in bf16 for stability
    bnb_4bit_use_double_quant=True,       # quantize the quantization constants
)

# ── Load model in 4-bit ─────────────────────────────────────
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b-hf",         # 70B model — fits on 1x A100 80GB!
    quantization_config=bnb_config,
    device_map="auto",
)

# ── Prepare for k-bit training ──────────────────────────────
# Casts layernorms to fp32, freezes base, enables gradient checkpointing
model = prepare_model_for_kbit_training(model)

# ── LoRA config (slightly higher rank for 70B) ──────────────
lora_config = LoraConfig(
    r=16,
    lora_alpha=32,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                     "gate_proj","up_proj","down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 13,631,488 || all params: 69,544,042,496 || trainable%: 0.0196

Merging and saving LoRA adapters

from peft import PeftModel
from transformers import AutoModelForCausalLM

# ── Save just the adapter (tiny!) ────────────────────────────
model.save_pretrained("./my-adapter")
# Creates adapter_model.safetensors (~17 MB for r=8 on 7B)
# and adapter_config.json

# ── Load adapter onto base model later ──────────────────────
base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")
model = PeftModel.from_pretrained(base, "./my-adapter")

# ── Merge adapter into base weights ─────────────────────────
# After this, the model is a standard transformer — no adapter overhead
merged = model.merge_and_unload()
merged.save_pretrained("./merged-model")

# ── Hot-swap adapters at serving time ────────────────────────
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf")

# Load multiple adapters
model = PeftModel.from_pretrained(base, "./adapter-medical", adapter_name="medical")
model.load_adapter("./adapter-legal", adapter_name="legal")
model.load_adapter("./adapter-code", adapter_name="code")

# Switch between them at runtime — near-zero overhead
model.set_adapter("medical")
output_medical = model.generate(**inputs)

model.set_adapter("legal")
output_legal = model.generate(**inputs)

LoRA from scratch in PyTorch

import torch
import torch.nn as nn
import math

class LoRALinear(nn.Module):
    """
    Drop-in replacement for nn.Linear with LoRA adaptation.

    Forward: h = W₀x + (α/r) · B @ A @ x
    - W₀ is frozen (no grad)
    - A: (r, in_features) — Kaiming uniform init
    - B: (out_features, r) — zero init
    """
    def __init__(self, base_linear: nn.Linear, r: int = 8, alpha: float = 16.0,
                 dropout: float = 0.0):
        super().__init__()
        self.base = base_linear
        self.base.weight.requires_grad_(False)      # freeze original
        if self.base.bias is not None:
            self.base.bias.requires_grad_(False)

        d_out, d_in = base_linear.weight.shape
        self.r = r
        self.scaling = alpha / r

        # LoRA matrices
        self.A = nn.Parameter(torch.empty(r, d_in))
        self.B = nn.Parameter(torch.zeros(d_out, r))  # zero init!
        nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))

        self.dropout = nn.Dropout(dropout) if dropout > 0 else nn.Identity()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        # Frozen base path
        h = self.base(x)
        # LoRA path: project down → project up → scale
        lora_out = self.dropout(x) @ self.A.T @ self.B.T * self.scaling
        return h + lora_out

    def merge(self) -> nn.Linear:
        """Merge LoRA weights back into base — returns standard nn.Linear."""
        with torch.no_grad():
            self.base.weight += self.scaling * (self.B @ self.A)
        return self.base


# ── Usage ────────────────────────────────────────────────────
original = nn.Linear(4096, 4096, bias=False)
lora_layer = LoRALinear(original, r=8, alpha=16)

# Count parameters
total = sum(p.numel() for p in lora_layer.parameters())
trainable = sum(p.numel() for p in lora_layer.parameters() if p.requires_grad)
print(f"Total: {total:,}  Trainable: {trainable:,}  ({100*trainable/total:.4f}%)")
# Total: 16,842,752  Trainable: 65,536  (0.3891%)

References

Seminal papers and key works referenced in this article.

1. Hu et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR, 2022. arXiv
2. Dettmers et al. "QLoRA: Efficient Finetuning of Quantized Language Models." NeurIPS, 2023. arXiv
3. Houlsby et al. "Parameter-Efficient Transfer Learning for NLP." ICML, 2019. arXiv
4. Li & Liang. "Prefix-Tuning: Optimizing Continuous Prompts for Generation." ACL, 2021. arXiv