LoRA, QLoRA, and the PEFT Landscape

A 70B model in bfloat16 weighs 140 GB before you load the optimizer. Full fine-tuning adds two fp32 gradient moments per parameter — another 560 GB, minimum — and you still need activations. A single run requires four to eight H100s at ~$3/hour each, plus the checkpoint storage per experiment. At that price point, most teams run three experiments before they give up and go back to prompting. LoRA changes that arithmetic enough to matter.

What LoRA is actually doing

The insight behind LoRA (Hu et al., 2021) is that the weight update during fine-tuning has low intrinsic rank. When you adapt a pretrained model to a narrower task, you're not learning an arbitrary matrix — you're making a directional shift in a much smaller subspace. Rather than finding that subspace implicitly by updating all of W, LoRA parameterizes it explicitly.

For a weight matrix W ∈ ℝ^(d×k), LoRA adds two matrices:

A ∈ ℝ^(r×k)   (initialized with Gaussian noise — the down-projection)
B ∈ ℝ^(d×r)   (initialized to zero — so ΔW = 0 at the start of training)

Effective weight: W + ΔW = W + (α/r) × BA

W stays frozen. Only A and B are trained. r is the rank, typically 8–64. α (alpha) is a scaling factor, conventionally set to 2r. Setting B=0 at init means the adapter contributes nothing at the start of training, so you're starting from exactly the pretrained model's behavior — no cold-start distribution shift.

The parameter count for a single layer's adapter is d×r + r×k. For Llama-3 8B with d=4096 and one q_proj of shape [4096, 4096], full fine-tuning trains 16.7M parameters per projection. LoRA at r=32 trains 4096×32 + 32×4096 = 262,144 — about 1.6% of that layer alone. Across all seven projection matrices in all 32 transformer layers — where GQA shrinks k_proj and v_proj to 4096→1024 and the MLP projections span 4096↔14336 — r=32 adds roughly 84M trainable parameters, or about 1% of the total. Adam stores two fp32 moments per trainable parameter, so your optimizer state is 84M × 4 bytes × 2 ≈ 670 MB instead of the 64 GB you would need for the full model.

flowchart TD
    INP["Input x"] --> W["Frozen W\n(d × k)"]
    INP --> A["Trainable A\n(r × k)"]
    A --> B["Trainable B\n(d × r)"]
    W --> SUM["Sum"]
    B --> SCALE["× α/r"]
    SCALE --> SUM
    SUM --> OUT["h = Wx + (α/r)BAx"]

    style W fill:#0e7490,color:#fff
    style A fill:#00e5ff,color:#0a0a0f
    style B fill:#00e5ff,color:#0a0a0f
    style SCALE fill:#00e5ff,color:#0a0a0f

{ "type": "lora", "title": "LoRA rank: trainable params, adapter VRAM, and quality delta", "rank": 32, "model": "7b" }

QLoRA: making 4-bit bases trainable

LoRA cuts optimizer memory. The base model itself is still the bottleneck. Llama-3 8B in bfloat16 takes 16 GB — you haven't even started training. QLoRA (Dettmers et al., 2023) stacks two tricks on top of LoRA to compress the frozen base.

NF4 quantization. The frozen weights are stored in 4-bit NormalFloat format, which assigns quantization buckets to match the expected normal distribution of neural network weights. This is more accurate than naive int4 for network weights — the buckets are denser in the high-probability center of the distribution. An 8B model that costs 16 GB in bfloat16 fits in roughly 4–5 GB in NF4.

Double quantization. The quantization constants themselves are quantized. Each NF4 block has a float32 quantization constant; QLoRA quantizes those constants to 8-bit, saving another ~0.37 bits per parameter. Small, but it adds up at billions of parameters.

Paged optimizers. When GPU VRAM runs low during a gradient spike, optimizer states page to CPU RAM instead of causing an OOM kill. The training run slows down during paging events but doesn't crash.

The LoRA adapters remain in bfloat16 throughout — they need full precision because their gradients are what you actually care about. The forward pass dequantizes each NF4 block to bfloat16 just-in-time for the computation, then discards the dequantized values. This costs some throughput (roughly 20–30% slower than native bfloat16 training) but the memory trade-off is overwhelmingly favorable.

A concrete cost data point: Llama-3 8B fine-tuned on 50K examples using QLoRA on a single A100 80GB takes approximately 6 hours and costs around $12 at cloud GPU rates. The equivalent full fine-tune would require 4–8 A100s and cost $100–300 for the same run. For most teams, QLoRA is the default starting point.

Rank, alpha, and which modules to target

These three choices determine most of the quality you get from a LoRA run. Get them wrong and you'll spend three experiments wondering why your fine-tune isn't working.

Rank (r)

Rank controls the expressiveness of the adapter. Higher rank → more parameters → more capacity to learn task-specific transformations → also more VRAM.

Practical defaults:

• r=16: Style tasks, format adherence, tone, domain vocabulary. The model already knows how to answer; you're shaping how it answers.
• r=32: General instruction tuning, chat fine-tunes, customer-support Q&A on proprietary knowledge. The default for most SFT work.
• r=64: Multi-turn coding, complex tool use, technical domains requiring new reasoning chains. Use this if r=32 plateaus before target performance.
• r=128+: Usually a sign you should benchmark full fine-tuning. Past this point the memory advantage shrinks and the quality ceiling of low-rank approaches may be the binding constraint.

Alpha (α)

Alpha scales the adapter output: the effective update is (α/r) × BAx. The ratio α/r is what matters. Setting α = 2r keeps this ratio at 2.0 regardless of rank, which makes adapter outputs comparably scaled when you sweep ranks. If you want to treat alpha and rank separately, the more principled approach from rsLoRA (Kalajdzievski, 2023) scales by 1/√r instead of 1/r, which improves training stability at high ranks.

Target modules

This is where most practitioners leave quality on the table.

The default in many tutorials is target_modules=["q_proj", "v_proj"]. The reasoning is that query and value projections drive most of the attention pattern change. True, but incomplete. The MLP layers — gate_proj, up_proj, down_proj — contain roughly two-thirds of a transformer's parameters and are where most factual and reasoning capabilities live. Leaving them frozen is fine for pure style tasks, but for any task involving new knowledge, domain reasoning, or changed output structure, you're leaving the most expressive parameters untouched.

Target all seven:

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",   # attention
        "gate_proj", "up_proj", "down_proj",         # MLP
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 83,886,080 || all params: 8,114,147,328 || trainable%: 1.03%

The VRAM difference between q+v only and all seven on a 7B model is typically under 1 GB. There is almost no reason to omit the MLP projections.

One case that looks like it demands full fine-tuning but doesn't: adding special tokens (chat-template delimiters, tool-call markers). New rows in the embedding matrix start as noise, and a low-rank adapter can't repair that — so after calling resize_token_embeddings, set modules_to_save=["embed_tokens", "lm_head"] in the LoraConfig to train those two matrices in full alongside the adapters. Skip this and the model silently mangles every new token at inference. The cost is checkpoint size: both matrices save whole, adding roughly 2 GB in bfloat16 for Llama-3 8B (2 × 128K vocab × 4096 dims) on top of the ~170 MB adapter.

A complete QLoRA training setup

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from trl import SFTTrainer, SFTConfig

# 1. Load in 4-bit NF4
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,          # double quantization
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)
model = prepare_model_for_kbit_training(model)  # enables gradient checkpointing

# 2. Wrap with LoRA
lora_config = LoraConfig(
    r=32,
    lora_alpha=64,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)
model = get_peft_model(model, lora_config)

# 3. Train
trainer = SFTTrainer(
    model=model,
    args=SFTConfig(
        output_dir="./outputs",
        num_train_epochs=2,
        per_device_train_batch_size=2,
        gradient_accumulation_steps=8,       # effective batch = 16
        learning_rate=2e-4,
        lr_scheduler_type="cosine",
        warmup_ratio=0.03,
        bf16=True,
        logging_steps=10,
        save_strategy="epoch",
        optim="paged_adamw_8bit",            # paged optimizer for QLoRA
    ),
    train_dataset=dataset,
)
trainer.train()

# 4. Save adapter only (100–300 MB, not the full 16 GB base)
model.save_pretrained("./my-adapter")

The paged_adamw_8bit optimizer is the QLoRA-specific choice — it quantizes the optimizer states to 8-bit and pages to CPU RAM during spikes. The combination of 4-bit weights, 8-bit optimizer, and gradient checkpointing is what makes this fit on a 24 GB card.

Merging adapters for inference

A LoRA adapter at inference time adds a matrix multiplication per layer on the forward pass. The overhead is small but nonzero — roughly 2–5% slower than a native forward pass. If you're serving at scale or have strict latency budgets, merge the adapter back into the base before deployment.

from peft import PeftModel
import torch

# Load base in bfloat16 — NOT quantized — for merging
base = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Meta-Llama-3-8B-Instruct",
    torch_dtype=torch.bfloat16,   # must match adapter dtype
    device_map="cpu",
)

model = PeftModel.from_pretrained(base, "./my-adapter")
merged = model.merge_and_unload()

# Save merged checkpoint (full 16 GB, runs at full speed)
merged.save_pretrained("./merged-model")

The critical detail: the base model must be in bfloat16 when you merge. Merging into an int4/NF4 quantized base forces a lossy dequantize → merge → requantize round trip whose rounding error erodes the adapter's learned deltas (PEFT warns you when you try). Always reload the base in bfloat16 first. The merged checkpoint is a full model and runs identically to a native fine-tune at inference.

If you're maintaining multiple adapters against the same base (one per customer, one per task), keep them separate and load them at request time with model.load_adapter(). The base model stays in memory; adapters hot-swap in milliseconds. This is a legitimate production pattern when adapter count is in the tens; past ~100 simultaneous adapters you're into multi-adapter serving territory (tools like punica or vLLM's lora support handle this).

2025–2026 variants worth knowing

The LoRA ecosystem has moved beyond the original paper. Three variants have earned their place in production thinking.

DoRA (Weight-Decomposed LoRA, Liu et al., 2024). Decomposes W into a magnitude vector m and a unit-norm direction W/||W|| before applying LoRA to the direction. The result is that DoRA at r=8 matches LoRA at r=32 on most benchmarks — you get the same quality at roughly one-fourth the trainable parameters. If your constraint is compute rather than VRAM, DoRA gets you there faster. QDoRA combines DoRA with 4-bit quantization of the base and is the emerging best practice when both memory and compute efficiency matter.

PiSSA (Principal Singular Values and Singular Vectors Adaptation, Meng et al., 2024). Initializes A and B from the top-r singular vectors of W rather than random noise. This gives the adapter a warmer starting point that's closer to the important directions in the pretrained weight. Training converges faster and final quality is usually slightly higher than standard LoRA at the same rank.

rsLoRA. A training-time fix: scales the adapter by α/√r instead of α/r. At high ranks the original scaling causes the adapter updates to grow too large relative to the base model's gradient scale. rsLoRA stabilizes this and allows higher ranks without degrading training dynamics. Most training frameworks now support it via a single config flag.

flowchart LR
    subgraph "Method comparison (r=32 unless noted)"
        L["LoRA r=32\n~84M params\nbaseline quality"]
        D["DoRA r=8\n~21M params\n≈LoRA r=32 quality"]
        P["PiSSA r=16\n~42M params\nfaster convergence"]
        RS["rsLoRA r=64\n~168M params\nstable at high rank"]
    end

    style L fill:#0e7490,color:#fff
    style D fill:#00e5ff,color:#0a0a0f
    style P fill:#a855f7,color:#fff
    style RS fill:#15803d,color:#fff

What breaks

Targeting only q and v projections

Quality will plateau below full fine-tune because two-thirds of the model's parameters — the MLP — never move. This is the single most common LoRA mistake in tutorials and starter code. The shortcut was originally motivated by attention-head analysis papers showing that q and v projections are most responsible for what information is retrieved. That's true at inference. At fine-tuning time, the MLP is where domain knowledge and reasoning patterns get updated.

Merging adapters into a quantized base

If you're running QLoRA, the base sits in NF4 during training. Calling merge_and_unload() on that quantized base forces a dequantize → merge → requantize round trip: each 4-bit block is dequantized, the adapter delta added, and the result requantized back to 4-bit — PEFT emits a rounding-error warning while it does this. The result stays quantized, and the requantization rounding eats into exactly the small deltas your fine-tune spent hours learning, so quality on the fine-tuned task measurably degrades. Always reload the base in bfloat16 before merging.

Using DPO without an SFT warm-up

This falls outside LoRA itself but affects everyone who runs LoRA for SFT before preference optimization. DPO (covered in Preference Optimization) adjusts the model's output distribution based on chosen/rejected pairs. If you apply DPO directly to a base model that hasn't been instruction-tuned first, the log-ratio objective diverges because the reference distribution is the raw pretraining mixture — far from any preference signal. Always SFT first, even briefly.

Low rank on data-rich high-complexity tasks

If you have 500K+ curated examples and a complex target domain (long-form structured reasoning, multi-turn agent tool use), LoRA at r=32 may genuinely not be expressive enough. Before blaming the method, try r=64 and DoRA. If you're still capped, benchmark full fine-tuning on a representative subset — sometimes the low-rank constraint is real. See The Decision Ladder for the cost analysis that makes this comparison concrete.

Catastrophic forgetting via overfitting

LoRA's smaller parameter count helps here — you're less likely to overwrite pretrained knowledge. But at very low ranks (r=4, r=8) on large datasets, the adapter can still overfit and the base's general capabilities degrade on held-out tasks. Use a dropout rate of 0.05–0.1 on the adapters, keep training to 2–3 epochs with cosine LR decay, and evaluate on a held-out domain distribution that isn't your fine-tuning target. If general benchmarks drop more than 3–5 points, you've over-indexed.

Benchmark contamination when using synthetic data

If you generated your training data using a frontier model (GPT-4o, Claude Opus, Gemini 2.5 Pro) and plan to evaluate on standard benchmarks, check whether those benchmarks appear in the generating model's training data. GSM8K, MMLU, and HumanEval are almost certainly in the pretraining mix of any 2025-vintage frontier model. The generating model's capabilities on contaminated benchmarks will transfer to your adapter without reflecting genuine learning. Cross-check synthetic prompt content against eval sets before training. This is covered in more depth in Synthetic Data Engineering.

The multi-adapter serving pattern

One underused production pattern: maintain a single base model instance and swap LoRA adapters per request. Customer A has a fine-tuned adapter for medical terminology; customer B has one for legal drafting. The 5–6 GB base stays loaded; adapters (~200 MB each) load on demand.

from peft import PeftModel

# Load base once
base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct",
                                             torch_dtype=torch.bfloat16)

# Register adapters by name
base = PeftModel.from_pretrained(base, "./adapter-medical", adapter_name="medical")
base.load_adapter("./adapter-legal", adapter_name="legal")

# At request time:
base.set_adapter("medical")
out = base.generate(...)

base.set_adapter("legal")
out = base.generate(...)

This works cleanly up to ~20–30 adapters before the management overhead warrants a proper inference server. vLLM's LoRA serving (--enable-lora, --max-loras) handles this with adapter caching and GPU memory management up to hundreds of concurrent adapters. The adapter file is small enough (200 MB) that it loads from disk in under a second on NVMe, so cold-load latency on first request is acceptable.

When to choose what

The memory math drives almost every PEFT decision. But there are quality and operational dimensions that the numbers alone don't capture.

Scenario	Recommended approach	Why
Style, tone, brand voice on any model size	LoRA r=16, q+k+v+o	Attention changes dominate; MLP not needed
General SFT / instruction tuning, <100K examples	QLoRA r=32, all 7 projections	Quality headroom, single-GPU feasibility
Complex reasoning, multi-turn tool use	QLoRA r=64 or DoRA r=16	High-rank or DoRA for reasoning tasks
Quality ceiling hit with LoRA	DoRA r=8–16	Better quality-per-parameter at lower rank
Serving N tenants from one base	LoRA r=16–32 + adapter hot-swap	Shared base, small per-tenant adapter
>500K curated examples, compute available	Full fine-tune in bfloat16	Low-rank constraint may cap quality
Memory at absolute premium (4–8 GB GPU)	GPTQ or AWQ base + LoRA inference adapters	Inference-time adapters on quantized base

The production reality in 2025–2026. Most teams shipping specialized models are using QLoRA for the training run and merging before deployment. The merge erases latency overhead; the QLoRA run makes the experiment cheap enough to iterate. A full-quality SFT experiment on a 7B model costs under $15 on a rented A100. That price makes it feasible to run five or ten experiments with different data mixes, which in practice matters more than squeezing the last 1% from a single perfect run.

If you're coming from the SFT and Instruction Tuning article, the message there about data quality applies here equally: a QLoRA run on 2,000 curated examples will beat a full fine-tune on 200,000 mediocre ones. The adapter method is not the binding constraint. The data is. Once you have the data, QLoRA makes the compute accessible; the rest is iteration.