# LoRA, QLoRA, and the PEFT Landscape > How low-rank adapters let you fine-tune a 70B model on a single GPU, with the rank, alpha, and target-module settings that actually matter in production. Canonical: https://ironclad.academy/ai-engineering/articles/lora-qlora-peft-parameter-efficient-fine-tuning | Difficulty: medium | Topics: Fine-Tuning, Post-Training, Quantization, LLM Internals Covers: Hugging Face, Meta, NVIDIA ## Key takeaways - Full fine-tuning a 7B model needs 100–120 GB VRAM; QLoRA does it on an RTX 4090 (24 GB) at roughly 90–95% quality retention. - Target all seven projection matrices — q, k, v, o, gate_proj, up_proj, down_proj — not just q and v, or you leave the MLP parameters frozen and cap quality on reasoning tasks. - Set alpha = 2 × rank as the default; use r=16 for style tasks, r=32 for general SFT, r=64 for complex multi-turn coding or tool use. - Merging LoRA adapters back into base weights at inference eliminates all latency overhead; a merged checkpoint runs identically to a full fine-tune. - DoRA at r=8 matches LoRA at r=32 on most benchmarks — if you are paying for compute, prefer DoRA over just cranking up the rank. - Merging adapters into an int4 quantized base forces a lossy requantization that degrades the fine-tuned behavior; always reload the base in bfloat16 before merging. > **In one line:** LoRA injects tiny trainable low-rank matrices into a frozen pretrained model, and QLoRA adds 4-bit quantization on top, together cutting fine-tuning VRAM requirements by 10–20× while keeping 90–95% of full-fine-tune quality. **The idea.** Full fine-tuning updates every weight in the model. For a 7B parameter model, that means 14 GB of weights in bfloat16, plus Adam's two fp32 gradient moments per parameter (another 56 GB), plus activations — you're looking at 100–120 GB before a single gradient step. LoRA sidesteps this entirely: freeze the base, inject two small matrices A and B whose product ΔW = BA approximates the weight update you would have computed, and train only those. The base model's knowledge stays intact; the adapter bends its outputs toward your task. QLoRA adds 4-bit NF4 quantization to the frozen base, cutting it from 14 GB to ~4–5 GB, making a 7B fine-tune feasible on consumer hardware. **Key points.** - LoRA trains ~0.1–1% of the original parameter count; optimizer memory drops from tens of GB to well under 1 GB. - QLoRA (4-bit base + bfloat16 adapters + paged optimizers) fits Llama-3 8B fine-tuning on an RTX 4090 (24 GB). - Rank `r` governs the expressiveness of the adapter — r=16 for style, r=32 for general SFT, r=64 for complex tasks; set `alpha = 2r`. - Target all seven projection matrices (q, k, v, o, gate, up, down), not just q+v. - LoRA adapters merge back into base weights at inference with zero latency cost. - 2025–2026 variants — DoRA, PiSSA, rsLoRA — improve per-parameter efficiency; DoRA at r=8 matches LoRA at r=32. **Memory math (back-of-the-envelope).** ``` Llama-3 8B in bfloat16: 8B × 2 bytes = 16 GB (weights) + 16 GB (Adam moments, 2× weights) + ~10 GB (activations, batch-dependent) = ~42 GB minimum, often 80+ GB with batch size > 1 Same with QLoRA (NF4 base + bfloat16 adapters, r=32, all 7 projections): 8B × 0.5 bytes (NF4) ≈ 4–5 GB (base, frozen) ~1% trainable params ≈ 84M × 2 bytes (bf16) + 84M × 2 bytes (8-bit Adam, 2 moments) ≈ 0.35 GB + paged optimizer: offloads overflow to CPU RAM + activations: ~4–6 GB = 10–12 GB on GPU, fits on RTX 4090 (24 GB) with room to spare Resulting adapters checkpoint: ~100–300 MB (r=32, all 7 modules, 7B model) Full checkpoint for comparison: ~14–16 GB ``` ```mermaid flowchart LR subgraph "Training" B["Frozen base weights\n(NF4, ~4-5 GB for 7B)"] --> F["Forward pass"] A["LoRA adapter\nBA (bfloat16, ~0.17 GB)"] --> F F --> L["Loss"] L -->|"gradient only through A, B"| A end subgraph "Inference" BM["Base weights"] --> MRG["Merged: W + αBA/r"] AM["Adapter"] --> MRG MRG --> OUT["Zero latency overhead"] end ``` **Key design decisions.** | Decision | Recommended choice | Why | | --- | --- | --- | | Base precision during training | NF4 (QLoRA) | 10–20× VRAM reduction vs bfloat16 | | Adapter dtype | bfloat16 | Enough range for gradient updates; fp16 risks underflow | | Target modules | All 7: q, k, v, o, gate, up, down | MLP matters for reasoning; q+v-only caps gains | | Rank | 16 / 32 / 64 by task complexity | Higher rank = more expressiveness = more memory | | Alpha | 2× rank | Keeps effective adapter scale stable across rank choices | | Merge at inference | Yes, always | Zero latency overhead; adapter checkpoints remain usable separately | | Adapter method | LoRA → DoRA for quality-per-param | DoRA r=8 ≈ LoRA r=32 quality | **If you have 60 seconds, say this.** "LoRA freezes the pretrained model and injects small paired A and B matrices into the attention and MLP projections — you train only those, which is 0.1–1% of parameters. QLoRA adds 4-bit quantization of the frozen base, getting the VRAM for a 7B fine-tune down from 80+ GB to under 15 GB. I target all seven projection matrices, set rank to 32 for general SFT with alpha at 64, and merge adapters back into the base weights before shipping — so there's zero inference overhead. For quality-per-parameter efficiency I'd switch to DoRA, which matches LoRA at a quarter of the rank." A 70B model in bfloat16 weighs 140 GB before you load the optimizer. Full fine-tuning adds two fp32 gradient moments per parameter — another 560 GB, minimum — and you still need activations. A single run requires four to eight H100s at ~$3/hour each, plus the checkpoint storage per experiment. At that price point, most teams run three experiments before they give up and go back to prompting. LoRA changes that arithmetic enough to matter. ## What LoRA is actually doing The insight behind LoRA ([Hu et al., 2021](https://arxiv.org/pdf/2106.09685)) is that the weight update during fine-tuning has low intrinsic rank. When you adapt a pretrained model to a narrower task, you're not learning an arbitrary matrix — you're making a directional shift in a much smaller subspace. Rather than finding that subspace implicitly by updating all of W, LoRA parameterizes it explicitly. For a weight matrix W ∈ ℝ^(d×k), LoRA adds two matrices: ``` A ∈ ℝ^(r×k) (initialized with Gaussian noise — the down-projection) B ∈ ℝ^(d×r) (initialized to zero — so ΔW = 0 at the start of training) Effective weight: W + ΔW = W + (α/r) × BA ``` W stays frozen. Only A and B are trained. `r` is the rank, typically 8–64. `α` (alpha) is a scaling factor, conventionally set to `2r`. Setting B=0 at init means the adapter contributes nothing at the start of training, so you're starting from exactly the pretrained model's behavior — no cold-start distribution shift. The parameter count for a single layer's adapter is `d×r + r×k`. For Llama-3 8B with d=4096 and one q_proj of shape [4096, 4096], full fine-tuning trains 16.7M parameters per projection. LoRA at r=32 trains 4096×32 + 32×4096 = 262,144 — about 1.6% of that layer alone. Across all seven projection matrices in all 32 transformer layers — where GQA shrinks k_proj and v_proj to 4096→1024 and the MLP projections span 4096↔14336 — r=32 adds roughly 84M trainable parameters, or about 1% of the total. Adam stores two fp32 moments per trainable parameter, so your optimizer state is 84M × 4 bytes × 2 ≈ 670 MB instead of the 64 GB you would need for the full model. ```mermaid flowchart TD INP["Input x"] --> W["Frozen W\n(d × k)"] INP --> A["Trainable A\n(r × k)"] A --> B["Trainable B\n(d × r)"] W --> SUM["Sum"] B --> SCALE["× α/r"] SCALE --> SUM SUM --> OUT["h = Wx + (α/r)BAx"] style W fill:#0e7490,color:#fff style A fill:#00e5ff,color:#0a0a0f style B fill:#00e5ff,color:#0a0a0f style SCALE fill:#00e5ff,color:#0a0a0f ``` [Interactive visualizer on the original page: lora — LoRA rank: trainable params, adapter VRAM, and quality delta] ## QLoRA: making 4-bit bases trainable LoRA cuts optimizer memory. The base model itself is still the bottleneck. Llama-3 8B in bfloat16 takes 16 GB — you haven't even started training. QLoRA ([Dettmers et al., 2023](https://arxiv.org/pdf/2305.14314)) stacks two tricks on top of LoRA to compress the frozen base. **NF4 quantization.** The frozen weights are stored in 4-bit NormalFloat format, which assigns quantization buckets to match the expected normal distribution of neural network weights. This is more accurate than naive int4 for network weights — the buckets are denser in the high-probability center of the distribution. An 8B model that costs 16 GB in bfloat16 fits in roughly 4–5 GB in NF4. **Double quantization.** The quantization constants themselves are quantized. Each NF4 block has a float32 quantization constant; QLoRA quantizes those constants to 8-bit, saving another ~0.37 bits per parameter. Small, but it adds up at billions of parameters. **Paged optimizers.** When GPU VRAM runs low during a gradient spike, optimizer states page to CPU RAM instead of causing an OOM kill. The training run slows down during paging events but doesn't crash. The LoRA adapters remain in bfloat16 throughout — they need full precision because their gradients are what you actually care about. The forward pass dequantizes each NF4 block to bfloat16 just-in-time for the computation, then discards the dequantized values. This costs some throughput (roughly 20–30% slower than native bfloat16 training) but the memory trade-off is overwhelmingly favorable. A concrete cost data point: Llama-3 8B fine-tuned on 50K examples using QLoRA on a single A100 80GB takes approximately 6 hours and costs around $12 at cloud GPU rates. The equivalent full fine-tune would require 4–8 A100s and cost $100–300 for the same run. For most teams, QLoRA is the default starting point. ## Rank, alpha, and which modules to target These three choices determine most of the quality you get from a LoRA run. Get them wrong and you'll spend three experiments wondering why your fine-tune isn't working. ### Rank (r) Rank controls the expressiveness of the adapter. Higher rank → more parameters → more capacity to learn task-specific transformations → also more VRAM. Practical defaults: - **r=16**: Style tasks, format adherence, tone, domain vocabulary. The model already knows how to answer; you're shaping how it answers. - **r=32**: General instruction tuning, chat fine-tunes, customer-support Q&A on proprietary knowledge. The default for most SFT work. - **r=64**: Multi-turn coding, complex tool use, technical domains requiring new reasoning chains. Use this if r=32 plateaus before target performance. - **r=128+**: Usually a sign you should benchmark full fine-tuning. Past this point the memory advantage shrinks and the quality ceiling of low-rank approaches may be the binding constraint. ### Alpha (α) Alpha scales the adapter output: the effective update is `(α/r) × BAx`. The ratio α/r is what matters. Setting α = 2r keeps this ratio at 2.0 regardless of rank, which makes adapter outputs comparably scaled when you sweep ranks. If you want to treat alpha and rank separately, the more principled approach from rsLoRA ([Kalajdzievski, 2023](https://arxiv.org/pdf/2312.03732)) scales by `1/√r` instead of `1/r`, which improves training stability at high ranks. ### Target modules This is where most practitioners leave quality on the table. The default in many tutorials is `target_modules=["q_proj", "v_proj"]`. The reasoning is that query and value projections drive most of the attention pattern change. True, but incomplete. The MLP layers — `gate_proj`, `up_proj`, `down_proj` — contain roughly two-thirds of a transformer's parameters and are where most factual and reasoning capabilities live. Leaving them frozen is fine for pure style tasks, but for any task involving new knowledge, domain reasoning, or changed output structure, you're leaving the most expressive parameters untouched. Target all seven: ```python from peft import LoraConfig, get_peft_model config = LoraConfig( r=32, lora_alpha=64, target_modules=[ "q_proj", "k_proj", "v_proj", "o_proj", # attention "gate_proj", "up_proj", "down_proj", # MLP ], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) model = get_peft_model(base_model, config) model.print_trainable_parameters() # trainable params: 83,886,080 || all params: 8,114,147,328 || trainable%: 1.03% ``` The VRAM difference between q+v only and all seven on a 7B model is typically under 1 GB. There is almost no reason to omit the MLP projections. One case that looks like it demands full fine-tuning but doesn't: adding special tokens (chat-template delimiters, tool-call markers). New rows in the embedding matrix start as noise, and a low-rank adapter can't repair that — so after calling `resize_token_embeddings`, set `modules_to_save=["embed_tokens", "lm_head"]` in the `LoraConfig` to train those two matrices in full alongside the adapters. Skip this and the model silently mangles every new token at inference. The cost is checkpoint size: both matrices save whole, adding roughly 2 GB in bfloat16 for Llama-3 8B (2 × 128K vocab × 4096 dims) on top of the ~170 MB adapter. ## A complete QLoRA training setup ```python import torch from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training from trl import SFTTrainer, SFTConfig # 1. Load in 4-bit NF4 bnb_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True, # double quantization ) model = AutoModelForCausalLM.from_pretrained( "meta-llama/Meta-Llama-3-8B-Instruct", quantization_config=bnb_config, device_map="auto", ) model = prepare_model_for_kbit_training(model) # enables gradient checkpointing # 2. Wrap with LoRA lora_config = LoraConfig( r=32, lora_alpha=64, target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"], lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", ) model = get_peft_model(model, lora_config) # 3. Train trainer = SFTTrainer( model=model, args=SFTConfig( output_dir="./outputs", num_train_epochs=2, per_device_train_batch_size=2, gradient_accumulation_steps=8, # effective batch = 16 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=True, logging_steps=10, save_strategy="epoch", optim="paged_adamw_8bit", # paged optimizer for QLoRA ), train_dataset=dataset, ) trainer.train() # 4. Save adapter only (100–300 MB, not the full 16 GB base) model.save_pretrained("./my-adapter") ``` The `paged_adamw_8bit` optimizer is the QLoRA-specific choice — it quantizes the optimizer states to 8-bit and pages to CPU RAM during spikes. The combination of 4-bit weights, 8-bit optimizer, and gradient checkpointing is what makes this fit on a 24 GB card. ## Merging adapters for inference A LoRA adapter at inference time adds a matrix multiplication per layer on the forward pass. The overhead is small but nonzero — roughly 2–5% slower than a native forward pass. If you're serving at scale or have strict latency budgets, merge the adapter back into the base before deployment. ```python from peft import PeftModel import torch # Load base in bfloat16 — NOT quantized — for merging base = AutoModelForCausalLM.from_pretrained( "meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16, # must match adapter dtype device_map="cpu", ) model = PeftModel.from_pretrained(base, "./my-adapter") merged = model.merge_and_unload() # Save merged checkpoint (full 16 GB, runs at full speed) merged.save_pretrained("./merged-model") ``` The critical detail: the base model must be in bfloat16 when you merge. Merging into an int4/NF4 quantized base forces a lossy dequantize → merge → requantize round trip whose rounding error erodes the adapter's learned deltas (PEFT warns you when you try). Always reload the base in bfloat16 first. The merged checkpoint is a full model and runs identically to a native fine-tune at inference. If you're maintaining multiple adapters against the same base (one per customer, one per task), keep them separate and load them at request time with `model.load_adapter()`. The base model stays in memory; adapters hot-swap in milliseconds. This is a legitimate production pattern when adapter count is in the tens; past ~100 simultaneous adapters you're into multi-adapter serving territory (tools like [punica](https://github.com/punica-ai/punica) or vLLM's lora support handle this). ## 2025–2026 variants worth knowing The LoRA ecosystem has moved beyond the original paper. Three variants have earned their place in production thinking. **DoRA (Weight-Decomposed LoRA, [Liu et al., 2024](https://arxiv.org/pdf/2402.09353)).** Decomposes W into a magnitude vector m and a unit-norm direction W/||W|| before applying LoRA to the direction. The result is that DoRA at r=8 matches LoRA at r=32 on most benchmarks — you get the same quality at roughly one-fourth the trainable parameters. If your constraint is compute rather than VRAM, DoRA gets you there faster. **QDoRA** combines DoRA with 4-bit quantization of the base and is the emerging best practice when both memory and compute efficiency matter. **PiSSA (Principal Singular Values and Singular Vectors Adaptation, [Meng et al., 2024](https://arxiv.org/pdf/2404.02948)).** Initializes A and B from the top-r singular vectors of W rather than random noise. This gives the adapter a warmer starting point that's closer to the important directions in the pretrained weight. Training converges faster and final quality is usually slightly higher than standard LoRA at the same rank. **rsLoRA.** A training-time fix: scales the adapter by `α/√r` instead of `α/r`. At high ranks the original scaling causes the adapter updates to grow too large relative to the base model's gradient scale. rsLoRA stabilizes this and allows higher ranks without degrading training dynamics. Most training frameworks now support it via a single config flag. ```mermaid flowchart LR subgraph "Method comparison (r=32 unless noted)" L["LoRA r=32\n~84M params\nbaseline quality"] D["DoRA r=8\n~21M params\n≈LoRA r=32 quality"] P["PiSSA r=16\n~42M params\nfaster convergence"] RS["rsLoRA r=64\n~168M params\nstable at high rank"] end style L fill:#0e7490,color:#fff style D fill:#00e5ff,color:#0a0a0f style P fill:#a855f7,color:#fff style RS fill:#15803d,color:#fff ``` ## What breaks ### Targeting only q and v projections Quality will plateau below full fine-tune because two-thirds of the model's parameters — the MLP — never move. This is the single most common LoRA mistake in tutorials and starter code. The shortcut was originally motivated by attention-head analysis papers showing that q and v projections are most responsible for what information is retrieved. That's true at inference. At fine-tuning time, the MLP is where domain knowledge and reasoning patterns get updated. ### Merging adapters into a quantized base If you're running QLoRA, the base sits in NF4 during training. Calling `merge_and_unload()` on that quantized base forces a dequantize → merge → requantize round trip: each 4-bit block is dequantized, the adapter delta added, and the result requantized back to 4-bit — PEFT emits a rounding-error warning while it does this. The result stays quantized, and the requantization rounding eats into exactly the small deltas your fine-tune spent hours learning, so quality on the fine-tuned task measurably degrades. Always reload the base in bfloat16 before merging. ### Using DPO without an SFT warm-up This falls outside LoRA itself but affects everyone who runs LoRA for SFT before preference optimization. DPO (covered in [Preference Optimization](/ai-engineering/articles/preference-optimization-dpo-simpo-orpo-grpo)) adjusts the model's output distribution based on chosen/rejected pairs. If you apply DPO directly to a base model that hasn't been instruction-tuned first, the log-ratio objective diverges because the reference distribution is the raw pretraining mixture — far from any preference signal. Always SFT first, even briefly. ### Low rank on data-rich high-complexity tasks If you have 500K+ curated examples and a complex target domain (long-form structured reasoning, multi-turn agent tool use), LoRA at r=32 may genuinely not be expressive enough. Before blaming the method, try r=64 and DoRA. If you're still capped, benchmark full fine-tuning on a representative subset — sometimes the low-rank constraint is real. See [The Decision Ladder](/ai-engineering/articles/fine-tuning-vs-rag-vs-prompting-decision-framework) for the cost analysis that makes this comparison concrete. ### Catastrophic forgetting via overfitting LoRA's smaller parameter count helps here — you're less likely to overwrite pretrained knowledge. But at very low ranks (r=4, r=8) on large datasets, the adapter can still overfit and the base's general capabilities degrade on held-out tasks. Use a dropout rate of 0.05–0.1 on the adapters, keep training to 2–3 epochs with cosine LR decay, and evaluate on a held-out domain distribution that isn't your fine-tuning target. If general benchmarks drop more than 3–5 points, you've over-indexed. ### Benchmark contamination when using synthetic data If you generated your training data using a frontier model (GPT-4o, Claude Opus, Gemini 2.5 Pro) and plan to evaluate on standard benchmarks, check whether those benchmarks appear in the generating model's training data. GSM8K, MMLU, and HumanEval are almost certainly in the pretraining mix of any 2025-vintage frontier model. The generating model's capabilities on contaminated benchmarks will transfer to your adapter without reflecting genuine learning. Cross-check synthetic prompt content against eval sets before training. This is covered in more depth in [Synthetic Data Engineering](/ai-engineering/articles/synthetic-data-engineering-llm-fine-tuning). ## The multi-adapter serving pattern One underused production pattern: maintain a single base model instance and swap LoRA adapters per request. Customer A has a fine-tuned adapter for medical terminology; customer B has one for legal drafting. The 5–6 GB base stays loaded; adapters (~200 MB each) load on demand. ```python from peft import PeftModel # Load base once base = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct", torch_dtype=torch.bfloat16) # Register adapters by name base = PeftModel.from_pretrained(base, "./adapter-medical", adapter_name="medical") base.load_adapter("./adapter-legal", adapter_name="legal") # At request time: base.set_adapter("medical") out = base.generate(...) base.set_adapter("legal") out = base.generate(...) ``` This works cleanly up to ~20–30 adapters before the management overhead warrants a proper inference server. vLLM's LoRA serving (`--enable-lora`, `--max-loras`) handles this with adapter caching and GPU memory management up to hundreds of concurrent adapters. The adapter file is small enough (200 MB) that it loads from disk in under a second on NVMe, so cold-load latency on first request is acceptable. ## When to choose what The memory math drives almost every PEFT decision. But there are quality and operational dimensions that the numbers alone don't capture. | Scenario | Recommended approach | Why | | --- | --- | --- | | Style, tone, brand voice on any model size | LoRA r=16, q+k+v+o | Attention changes dominate; MLP not needed | | General SFT / instruction tuning, <100K examples | QLoRA r=32, all 7 projections | Quality headroom, single-GPU feasibility | | Complex reasoning, multi-turn tool use | QLoRA r=64 or DoRA r=16 | High-rank or DoRA for reasoning tasks | | Quality ceiling hit with LoRA | DoRA r=8–16 | Better quality-per-parameter at lower rank | | Serving N tenants from one base | LoRA r=16–32 + adapter hot-swap | Shared base, small per-tenant adapter | | >500K curated examples, compute available | Full fine-tune in bfloat16 | Low-rank constraint may cap quality | | Memory at absolute premium (4–8 GB GPU) | GPTQ or AWQ base + LoRA inference adapters | Inference-time adapters on quantized base | **The production reality in 2025–2026.** Most teams shipping specialized models are using QLoRA for the training run and merging before deployment. The merge erases latency overhead; the QLoRA run makes the experiment cheap enough to iterate. A full-quality SFT experiment on a 7B model costs under $15 on a rented A100. That price makes it feasible to run five or ten experiments with different data mixes, which in practice matters more than squeezing the last 1% from a single perfect run. If you're coming from the [SFT and Instruction Tuning](/ai-engineering/articles/sft-instruction-tuning-data-quality) article, the message there about data quality applies here equally: a QLoRA run on 2,000 curated examples will beat a full fine-tune on 200,000 mediocre ones. The adapter method is not the binding constraint. The data is. Once you have the data, QLoRA makes the compute accessible; the rest is iteration. ## Frequently asked questions Q: What is LoRA and how does it reduce fine-tuning memory? A: LoRA (Low-Rank Adaptation) freezes the pretrained model weights and injects small trainable low-rank matrices into the attention and MLP projection layers. Instead of updating all W parameters, it trains two matrices A and B where the rank r is far smaller than the full dimension — typically r=16–64 versus d=4096. The result is ~0.1–1% of the original parameter count being trained, which cuts the optimizer-state memory (Adam stores two fp32 moments per parameter, ~56 GB for a 7B model) to well under 1 GB for the adapters alone. Q: What is QLoRA and how much GPU memory does it actually use? A: QLoRA quantizes the frozen base model to 4-bit NF4 precision (using double quantization) while keeping the trainable LoRA adapters in bfloat16. A Llama-3 8B base model that would need ~16 GB in fp16 fits in roughly 4–5 GB with NF4 quantization. A full QLoRA fine-tune with optimizer state and activations on Llama-3 8B fits on an RTX 4090 (24 GB) or a single A100 80GB for runs up to ~50K examples in about 6 hours at a cost near $12. Q: What LoRA rank and alpha should I use? A: Use r=16 for style or format tasks (low conceptual distance from base behavior), r=32 for general instruction tuning and chat, and r=64 for complex domains like multi-turn coding or tool use. Set alpha equal to 2× rank — so alpha=32 for r=16, alpha=64 for r=32. A higher alpha scales the adapter output up, compensating for the low-rank initialization. If you find yourself pushing past r=128, consider whether you actually need full fine-tuning instead. Q: Which projection matrices should I target with LoRA? A: Target all seven projection matrices: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj. The common shortcut of targeting only q_proj and v_proj cuts VRAM modestly but leaves the MLP layers (gate, up, down) completely frozen, which caps quality gains on any task requiring new reasoning patterns or domain knowledge. The VRAM difference between q+v only versus all seven is typically under 1 GB on a 7B model. Q: When should I use LoRA versus full fine-tuning? A: Use LoRA whenever you are budget-constrained (single-GPU), need to maintain multiple adapters per task (LoRA adapters are ~50–300 MB versus a full 14 GB checkpoint), or plan to share a base model across teams. Use full fine-tuning when you have very large high-quality datasets (>500K examples), are changing the base model architecture or replacing the vocabulary wholesale (adding a few special tokens is handled in LoRA by setting modules_to_save for embed_tokens and lm_head), or have measured that LoRA quality ceilings out below your target threshold. Most production SFT work in 2025–2026 is QLoRA. Q: What is DoRA and is it better than LoRA? A: DoRA (Weight-Decomposed Low-Rank Adaptation) decomposes the weight matrix into a magnitude vector and a directional component, then applies LoRA only to the direction. The practical benefit is that DoRA at r=8 matches LoRA at r=32 quality on most benchmarks, cutting the trainable parameter count to roughly a quarter for the same result. QDoRA (DoRA plus 4-bit quantization) is the emerging best practice when memory is the binding constraint and quality matters more than speed of experimentation.