# Quantization Demystified: GPTQ, AWQ, GGUF, and FP8 With Real Tradeoffs > A practical comparison of GPTQ, AWQ, GGUF, and FP8 quantization formats — memory math, quality loss, hardware targets, and when each format actually wins in production. Canonical: https://ironclad.academy/ai-engineering/articles/quantization-gptq-awq-gguf-fp8 | Difficulty: medium | Topics: Quantization, Inference, Serving, Cost Optimization Covers: NVIDIA, Hugging Face, Meta ## Key takeaways - AWQ is the default 4-bit choice for 2026: faster to produce than GPTQ, better quality on reasoning tasks, and widely supported in vLLM and SGLang. - FP8 is the lossless alternative on H100/H200/B200 hardware — near-BF16 quality at half the VRAM, with roughly 1.8× BF16 throughput (~1,600 tok/s aggregate decode for a 70B on one H100). - GGUF is for CPU and hybrid CPU+GPU inference only — do not run it in a multi-user production GPU server where continuous batching matters. - A 70B model in BF16 needs ~140 GB VRAM; at AWQ INT4 that drops to ~38 GB — a single A100-80GB with KV-cache headroom, versus four GPUs at BF16. - Below 3-bit quantization, quality degrades faster than the memory savings justify — stay at INT4 as your floor for anything user-facing. - Quantization is not free: calibration data quality matters for GPTQ, salient weight detection matters for AWQ, and kernel availability determines whether theoretical gains translate to real throughput. > **In one line:** Quantization cuts LLM VRAM by 2-4x by reducing weight precision from BF16 to 8 or 4 bits — the formats differ in how they protect quality, what hardware they target, and how expensive they are to produce. **The idea.** An unquantized 70B model at BF16 needs ~140 GB of VRAM before you generate a single token. That means four A100-80GB GPUs just to load the weights — before batching, KV cache, or any concurrency headroom. Quantization trades a small, measurable amount of model quality for a large, predictable drop in memory: INT4 weights for a 70B model fit in ~38 GB — a single A100-80GB instead of four, roughly a 4× cut in the GPU bill per replica. The four formats that actually ship production traffic in 2026 — GPTQ, AWQ, GGUF, and FP8 — differ in how they protect quality during that compression, which hardware they run on, and how long they take to produce. **Key points.** - **AWQ** identifies the ~1% of weights with high activation salience and protects them; produces best 4-bit reasoning quality, takes tens of minutes to run. - **GPTQ** minimizes per-layer reconstruction error using second-order Hessian information; hours to compute, still widely distributed on Hugging Face. - **GGUF** is llama.cpp's multi-level format (Q2_K through Q8_0), the only practical path for CPU or hybrid CPU+GPU inference; the Ollama default. - **FP8** uses hardware-native 8-bit float arithmetic on H100/H200/B200 GPUs — near-BF16 quality, 2× weight compression, fastest throughput at datacenter scale. - Quality loss at 4-bit is ~1-2% on standard benchmarks; below 3 bits it accelerates sharply. **Memory math (back-of-envelope).** ``` Llama 3.1 70B parameter count: ~70 billion Bytes per weight: BF16 → 2 bytes → 70B × 2 = 140 GB FP8 → 1 byte → 70B × 1 = 70 GB (+ ~5 GB for FP16 activation buffers) INT4 → 0.5 byte → 70B × 0.5 = 35 GB (+ overhead for group scales ≈ 38 GB total) INT2 → 0.25 byte → 70B × 0.25 = 17.5 GB (but quality is badly degraded) Note: this is weights only. Add KV cache on top. KV cache at BF16, 70B (8 kv_heads, 128 head_dim, 80 layers): ≈ 0.32 MB per token per request (see /articles/kv-cache-gpu-memory-math) At 32 concurrent × 4K context: 0.32 × 32 × 4096 ≈ 42 GB additional ``` **Format comparison at a glance.** | Format | Target hardware | Precision | Quality loss (vs BF16) | Time to quantize | Throughput relative | |---|---|---|---|---|---| | BF16 baseline | A100 / H100 | 16-bit | — | — | 1× | | FP8 | H100 / H200 / B200 / Ada (4090, L4, L40S) / RTX 5090 | 8-bit float | ~0% (near-zero) | Minutes | ~1.8× | | AWQ INT4 | Any CUDA GPU | 4-bit int (mixed) | ~1-2% | Tens of minutes | ~2× weight load | | GPTQ INT4 | Any CUDA GPU | 4-bit int | ~1-3% | Hours | ~2× weight load | | GGUF Q4_K_M | CPU / Apple Silicon / hybrid | 4-bit mixed | ~2-3% | Minutes (convert) | ~0.7× on GPU | ```mermaid flowchart LR BF16["BF16 baseline\n140 GB for 70B"] --> FP8["FP8\n~70 GB\nnear-lossless\nFP8-native GPUs"] BF16 --> AWQ["AWQ INT4\n~38 GB\n1-2% quality drop\nany CUDA GPU"] BF16 --> GPTQ["GPTQ INT4\n~38 GB\n1-3% quality drop\nany CUDA GPU"] BF16 --> GGUF["GGUF Q4_K_M\n~40 GB\n2-3% quality drop\nCPU / Apple Silicon"] style BF16 fill:#0e7490,color:#fff style FP8 fill:#a855f7,color:#fff style AWQ fill:#00e5ff,color:#0a0a0f style GPTQ fill:#15803d,color:#fff style GGUF fill:#ffaa00,color:#0a0a0f ``` **If you have 60 seconds, say this.** "Quantization compresses model weights so they need less GPU memory. At 4-bit INT4 a 70B model drops from 140 GB to ~38 GB. The three main approaches for CUDA GPUs are: GPTQ (older, slower to produce, widely available on Hugging Face), AWQ (faster to produce, better reasoning quality, the 2026 default), and FP8 (hardware-native on H100-class GPUs, near-zero quality loss). GGUF is for CPU and local inference only — Ollama uses it. Pick FP8 if you have H100/H200 hardware. Pick AWQ if you need 4-bit on older GPUs. Use GGUF only for local or edge deployments. Stay at 4-bit as your floor; below 3 bits quality falls off a cliff." A 70B model in BF16 costs you $14.40/hour on a four-A100 node on AWS (four A100-80GB at $3.60/GPU/hour, illustrative mid-2026 pricing). Run that continuously and you're at ~$10,000/month before you've processed a single user request. Cut to a single GPU with INT4 quantization and the monthly bill drops roughly 4× — while the model still passes most evals within 2 percentage points of its original score. That is the core premise of quantization, and it is real. The complication is that "INT4" is not one thing: GPTQ, AWQ, GGUF, and FP8 each make different tradeoffs, require different hardware, and produce different throughput characteristics. Picking the wrong one for your stack doesn't just cost money — it can produce a model that looks fine on a benchmark and quietly regresses on the one task your users actually care about. ## What quantization is actually doing Floating-point weights take 16 bits each in BF16. Quantization maps them to a smaller set of discrete values — 256 values at INT8, 16 values at INT4. The challenge is that not all weights are equal: a handful of them have large activation magnitudes and disproportionately influence the output. Compress those naively and you get visible quality loss. Protect them and you can push everything else to 4 bits without the model noticing much. The three mechanisms that matter: **Per-group scaling.** Instead of one scale factor per tensor (too coarse), use one scale per group of 128 weights. This means a 4-bit weight is decoded as `fp16_value = int4_weight × per_group_scale + per_group_zero_point`. The scales are stored in FP16, adding roughly 12% overhead over pure 4-bit storage, but group sizes of 64 or 128 are the sweet spot where quality holds. **Mixed-precision weight protection.** Some formats identify individual outlier weights and store them at higher precision (FP16 or FP8) alongside the quantized weights. This is computationally more expensive but matters for models where activation outliers are concentrated in specific attention heads. **Hardware arithmetic.** On H100/H200/B200, the tensor cores natively execute FP8 matrix multiplications at full hardware speed. INT4 kernels on older NVIDIA GPUs must unpack weights at runtime — still fast, but not hardware-native. ## GPTQ: the original, still everywhere [GPTQ](https://arxiv.org/abs/2210.17323) (2022, Frantar et al.) is the oldest widely-used post-training quantization method for LLMs. It processes each layer sequentially and minimizes the layer-wise reconstruction error using second-order Hessian information — essentially finding the INT4 values that make the quantized layer's output match the original layer's output as closely as possible on a calibration dataset. ```mermaid flowchart TD W["Full-precision weight matrix W\n(BF16)"] --> CAL["Calibration forward pass\n(128–512 calibration samples)"] CAL --> H["Compute Hessian H\n(second-order info about sensitivity)"] H --> OPT["Layer-wise quantization\n(solve: min ||WX - Ŵ_q X||² )"] OPT --> QW["Quantized weights Ŵ_q\n(INT4 + FP16 group scales)"] style W fill:#0e7490,color:#fff style QW fill:#00e5ff,color:#0a0a0f style H fill:#a855f7,color:#fff ``` The practical results: GPTQ at 4-bit with group size 128 achieves perplexity within 0.1–0.5 points of the BF16 original on standard language modeling benchmarks. On task-specific evals like MMLU the gap is typically 1-3 percentage points. The quality is good — not great for complex reasoning, adequate for most production use cases. The cost is time and calibration data. Quantizing a 70B model with GPTQ takes roughly 4-8 hours on a single A100. The calibration dataset matters: using C4 (the default) works well for general tasks, but if your workload is code generation, using code-heavy calibration data produces meaningfully better results. GPTQ is the dominant format on Hugging Face Hub because it has been around the longest. If you need a quantized model for a recently released open-weight model, GPTQ is usually available within days of the BF16 release, produced by community members using tools like [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ). vLLM and SGLang both support GPTQ natively. The throughput story for GPTQ is that weights are stored as INT4 but the GPU executes matrix multiplications after dequantizing to FP16 on the fly. This makes GPTQ memory-bandwidth-efficient (loading 4-bit weights takes half the time of BF16) but not compute-efficient (the actual multiply-add is FP16). For decode-phase inference — which is always memory-bandwidth-bound, as the [prefill vs decode article](/ai-engineering/articles/prefill-vs-decode-phases) explains — this is exactly the bottleneck you want to reduce. ## AWQ: better quality, faster to produce **AWQ (Activation-aware Weight Quantization)** takes a different angle. Rather than minimizing per-layer reconstruction error, it observes which weights have large corresponding input activations and protects those weights — keeping them at FP16 or scaling them more carefully before quantization. The insight: LLM weights have activation outliers concentrated in a small fraction of channels (~0.1-1%). If you quantize those channels to INT4 with the same group scaling as everything else, you lose disproportionate quality. AWQ identifies these salient weights by measuring activation magnitudes on a calibration set, then scales the entire weight channel down by a per-channel factor before quantization and compensates by scaling activations up — effectively applying a smoother quantization to the sensitive channels. ```python # Illustrative AWQ quantization with autoawq (not full production config) from awq import AutoAWQForCausalLM from transformers import AutoTokenizer model_path = "meta-llama/Meta-Llama-3.1-70B-Instruct" quant_path = "llama-3.1-70b-awq-int4" model = AutoAWQForCausalLM.from_pretrained(model_path, torch_dtype="auto") tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True) quant_config = { "zero_point": True, "q_group_size": 128, # group size for per-group scaling "w_bit": 4, # 4-bit weights "version": "GEMM", # GEMM kernel variant (vs GEMV for small batch) } # Calibration data — domain matters; use samples representative of your workload model.quantize(tokenizer, quant_config=quant_config) model.save_quantized(quant_path) tokenizer.save_pretrained(quant_path) ``` The practical advantage over GPTQ: AWQ runs in tens of minutes rather than hours, and the quality on reasoning-heavy benchmarks is consistently better. A 2024 comparison of AWQ vs GPTQ on Llama-2 70B showed AWQ with 0.3-0.5 percentage points higher MMLU than GPTQ at the same bit-width, with a larger gap on GSM8K (math reasoning). As of 2026, AWQ is the recommended default for new 4-bit production deployments. Both vLLM and SGLang load AWQ models directly: ```python # vLLM serving with AWQ from vllm import LLM, SamplingParams llm = LLM( model="llama-3.1-70b-awq-int4", quantization="awq", gpu_memory_utilization=0.85, # leave headroom for KV cache max_model_len=8192, ) params = SamplingParams(temperature=0.7, max_tokens=512) outputs = llm.generate(["Explain quantization to an engineer."], params) ``` One gotcha: AWQ GEMM kernels are optimized for batch sizes ≥ 8. For single-request or very small batch inference (a chatbot with one user), the GEMV variant is faster. vLLM selects this automatically, but if you're calling kernels directly, pick the right one. ## GGUF: the CPU-and-local-inference format **GGUF** is the file format used by [llama.cpp](https://github.com/ggerganov/llama.cpp), which supports a range of quantization levels under a unified naming scheme: Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and a few others. The letter suffix (K, M, S) refers to variant strategies within each bit-width. Q4_K_M is the practical default — 4-bit with medium-quality K-quant scaling. ```mermaid flowchart LR Q2["Q2_K\n~26 GB for 70B\nStrong degradation\nSurvival mode only"] Q4["Q4_K_M\n~40 GB for 70B\n2-3% quality drop\nOllama default"] Q6["Q6_K\n~58 GB for 70B\n<1% quality drop\nCPU quality sweetspot"] Q8["Q8_0\n~70 GB for 70B\n~0% quality drop\nPractically lossless"] Q2 --> Q4 --> Q6 --> Q8 style Q2 fill:#ff2e88,color:#fff style Q4 fill:#ffaa00,color:#0a0a0f style Q6 fill:#15803d,color:#fff style Q8 fill:#0e7490,color:#fff ``` GGUF's primary advantage is hardware flexibility. llama.cpp can run on CPU, Apple Silicon (using Metal), NVIDIA GPUs, AMD GPUs, and mixed CPU+GPU setups where part of the model offloads to system RAM. For a developer running a 7B model on an M3 MacBook Pro or a team deploying to edge hardware without a GPU, GGUF is the only practical option. Ollama uses GGUF as its native format. Pull a model with `ollama pull llama3.1:70b` and you're getting the Q4_K_M GGUF variant. The tradeoff on pure NVIDIA GPU serving: GGUF kernels in llama.cpp are slower than GPTQ or AWQ kernels in vLLM for multi-user batch workloads. The gap is meaningful — roughly 20-30% lower throughput at batch sizes above 8 on identical hardware. llama.cpp does implement continuous batching (and Ollama exposes parallel request slots), but the scheduler, memory management, and high-batch kernels are far less efficient than vLLM's PagedAttention stack. Running GGUF/Ollama as your production inference server for multiple concurrent users means leaving a large fraction of your GPU's throughput on the table. For local dev or single-user internal tools, this is fine. For a multi-user product, it is not. [Interactive visualizer on the original page: kv-cache — VRAM breakdown: 70B model at BF16 (weights + KV cache at 16 concurrent requests)] ## FP8: the datacenter default for H100+ hardware FP8 is categorically different from the INT4 approaches above. It uses 8-bit floating-point representation — for inference, both weights and activations use NVIDIA's E4M3 variant; the wider-exponent E5M2 is reserved for training gradients (and shows up as an optional KV-cache dtype) — which retains a floating-point exponent. The dynamic range is close to BF16, which means quantization error is dramatically smaller than INT4. NVIDIA H100, H200, B200, the Ada Lovelace generation (RTX 4090, L4, L40S), and the RTX 5090 have hardware tensor cores that execute FP8 matrix multiplications natively. This is not a software emulation — the arithmetic runs at full hardware speed, and the throughput gain comes from loading half the bytes from HBM per multiply-accumulate. On H100 SXM5, FP8 inference achieves roughly 1.8× the BF16 throughput on decode-bound workloads. Quality-wise, FP8 is almost indistinguishable from BF16. Published benchmarks on Llama 3.1 70B show less than 0.1 percentage point MMLU difference. For tasks that are already within BF16's ability, FP8 is essentially lossless. This makes it the natural first choice on datacenter hardware. ```python # vLLM with FP8 (H100/H200 required) from vllm import LLM llm = LLM( model="meta-llama/Meta-Llama-3.1-70B-Instruct", quantization="fp8", # hardware-native on H100/H200 kv_cache_dtype="fp8", # also quantize KV cache to FP8 gpu_memory_utilization=0.90, ) # FP8 KV cache halves KV memory, doubling concurrent request capacity # See /articles/kv-cache-gpu-memory-math for the full calculation ``` The `kv_cache_dtype="fp8"` line above is worth pausing on. The [KV cache article](/ai-engineering/articles/kv-cache-gpu-memory-math) covers the memory formula in detail, but the short version: FP8 KV cache on a 70B model at 32 concurrent × 8K context saves ~42 GB compared to BF16 KV cache (0.32 MB/token × 32 × 8192 ≈ 84 GB, halved). That is often more impactful than the weight compression itself at moderate context lengths. FP8 quantization of weights is fast — models like Llama 3 70B quantize in minutes using NVIDIA's `modelopt` toolkit or vLLM's built-in quantization — and the calibration dataset matters less than for INT4 because the quantization error is so small to begin with. The constraint: native FP8 compute needs FP8 tensor cores — Hopper (H100/H200), Blackwell, or Ada (RTX 4090, L4, L40S). On Ampere (A100 and older), vLLM falls back to weight-only FP8 via Marlin kernels: you keep the halved weight memory and bandwidth (often still faster than BF16 on memory-bound decode), but compute upconverts to FP16 and you get none of the FP8 tensor-core throughput. If your stack is A100 or older and you want real compression wins, AWQ INT4 is the stronger call. ## The quantization decision tree ```mermaid flowchart TD START["What hardware?"] --> H100["FP8-native: H100 / H200 / B200\nAda (4090, L4, L40S) / RTX 5090"] START --> A100["A100 / A10G / older NVIDIA"] START --> CPU["CPU or Apple Silicon\nor hybrid CPU+GPU"] H100 --> FP8_REC["Use FP8\nnear-lossless, fastest throughput\nvLLM fp8 or modelopt"] A100 --> MULTI{"Multi-user\nproduction server?"} MULTI -->|"Yes"| AWQ_REC["Use AWQ INT4\nbest quality-per-bit on CUDA\nautoawq + vLLM/SGLang"] MULTI -->|"No (local dev / single user)"| GGUF_OK["GGUF Q4_K_M is fine\nOllama or llama.cpp"] CPU --> GGUF_REC["Use GGUF Q4_K_M\n(or Q6_K for higher quality)\nlllama.cpp / Ollama"] style FP8_REC fill:#a855f7,color:#fff style AWQ_REC fill:#00e5ff,color:#0a0a0f style GGUF_REC fill:#ffaa00,color:#0a0a0f style GGUF_OK fill:#15803d,color:#fff ``` GPTQ doesn't appear prominently in this tree because in 2026 AWQ is strictly better for new deployments: better quality, faster to produce, equally supported. GPTQ remains relevant because it dominates the existing catalog of quantized models on Hugging Face Hub — if a model you need is only available as GPTQ, use it. If you're producing the quantization yourself, use AWQ. ## Worked numbers: serving cost at three quantization levels Suppose you're pricing a single serving replica for a 70B model (requests averaging 1K input + 500 output tokens). One replica at these throughputs handles on the order of 100-190 requests/minute; scale the fleet linearly for more — the per-token economics below are what carry. ``` Hardware: NVIDIA A100-80GB, $3.60/GPU/hour (illustrative, mid-2026 on-demand AWS) BF16 baseline: Weights: 140 GB → 2 A100s (with 20 GB left for KV cache — tight) More realistic: 4 A100s for any reasonable concurrency At 4 GPUs: $14.40/hour = ~$10,400/month Throughput: ~800 tok/s decode on 4 A100s (memory-bandwidth-bound) AWQ INT4: Weights: ~38 GB → fits on 1 A100 with 42 GB KV cache headroom At 1 GPU: $3.60/hour = ~$2,600/month Throughput: ~900 tok/s decode on 1 A100 (less weight bytes to load per step) Cost per million output tokens: $3.60 / (900 × 3600 / 1M) = ~$1.11/M tok FP8 on H100-80GB ($8.00/hour illustrative): Weights: ~72 GB → 1 H100 with 8 GB remaining Better: 1 H100 with FP8 KV cache for 50+ concurrent requests At 1 H100: $8.00/hour = ~$5,760/month Throughput: ~1,600+ tok/s decode (H100 HBM bandwidth + FP8 tensor cores) Cost per million output tokens: $8.00 / (1600 × 3600 / 1M) = ~$1.39/M tok Summary: AWQ on A100 = 4× cost reduction vs BF16 baseline (on same A100 hardware) FP8 on H100 = 1.8× throughput vs BF16 on same H100, ~45% lower cost vs H100+BF16 ``` The conclusion from this math: if you're on A100 hardware, AWQ INT4 is the single highest-impact change you can make to inference cost. If you're upgrading to H100 anyway, FP8 is the obvious format — the quality improvement over AWQ justifies the format change even if the cost numbers come out similar. ## What breaks **Quality collapse below 3 bits.** The Q2_K GGUF and similar 2-bit formats sound appealing on paper (a 70B model in ~26 GB — K-quants store scales, so "2-bit" really means ~2.6-3.4 bits per weight). In practice, 2-bit quantization destroys enough numerical precision that the model confuses basic facts, loses coherence on long outputs, and fails structured tasks. The perplexity numbers look acceptable on benchmarks but real task performance falls off a cliff. Stay at 4-bit as your floor for anything user-facing. **Calibration data mismatch.** GPTQ's Hessian approximation and AWQ's salient weight detection both depend on calibration data. Using generic Wikipedia/C4 calibration for a code-generation model means the quantization wasn't optimized for your actual distribution. The fix: use 128-512 samples from your target domain. For a coding assistant, that means code. The quality difference between matched and mismatched calibration at 4-bit can be 2-4 percentage points on task-specific evals. **GGUF in production GPU servers.** Running Ollama (GGUF-backed) as a multi-user production inference server is the most common mistake teams make when transitioning from local prototyping to deployment. Ollama does batch concurrent requests (via llama.cpp's parallel slots), but its scheduling and memory management are far less efficient at high concurrency than vLLM's continuous batching with PagedAttention. The throughput difference is not 10% — it is often 3-10× at 50+ concurrent users. See the [continuous batching article](/ai-engineering/articles/continuous-batching-scheduling) for the mechanics. **Kernel availability for new models.** The throughput gains from quantization depend on efficient GPU kernels, not just the reduced memory footprint. When a new model architecture ships, the AWQ/GPTQ kernels in vLLM take a few weeks to be validated. In that window, quantized inference may fall back to unoptimized paths and actually be slower than BF16. Check the vLLM model support table before assuming quantization will speed things up on a brand-new architecture. **FP8 on non-FP8-native hardware.** If you run `quantization="fp8"` in vLLM on an A100, it falls back to weight-only FP8 (Marlin kernels) with a logged warning — you keep the memory savings and often some decode speedup, but none of the native FP8 compute throughput the H100 benchmarks advertise. Miss the warning and you'll conclude FP8 is broken when it's actually running in a degraded mode. Know which mode your hardware gets before you benchmark. **KV cache precision mismatch.** Quantizing weights to INT4 while leaving the KV cache at BF16 leaves the bigger memory consumer untouched at high concurrency. Conversely, quantizing the KV cache to FP8 when your weights are still BF16 on an H100 is often the higher-value move — KV cache can exceed weight memory at moderate context lengths and batch sizes. Consider weight quantization and KV cache quantization independently. (The interactive widget above makes this concrete for a 70B model.) ## When to use what in practice **New deployment, H100/H200/B200:** Start with FP8. Run vLLM with `quantization="fp8"` and `kv_cache_dtype="fp8"`. Quality is near-BF16, throughput is the highest available. Only reach for INT4 if you need to fit a larger model onto the same GPU count. **Existing A100 or A10G fleet:** AWQ INT4 is the call. Produce the quantized model with `autoawq`, push to your model registry, serve with vLLM or SGLang. Expect ~2× weight-loading speedup on decode-bound inference and a significant reduction in GPU count required. **Local development or internal tooling with a single user:** GGUF with Ollama. It is the fastest path from model name to running inference and requires zero serving infrastructure. Use Q4_K_M for the quality/size balance, Q6_K if you have the VRAM and care about accuracy. Don't run it for anything requiring concurrent requests at scale. **Existing GPTQ model from Hugging Face, no re-quantization budget:** Use it. GPTQ is supported in vLLM and SGLang and the quality is good enough for most tasks. If quality matters and you have time, re-quantize to AWQ — the improvement on reasoning tasks is real. If you don't have time, ship the GPTQ model. **Model smaller than 13B:** At 7-8B parameters, BF16 fits in a single A100-80GB with ample KV cache headroom. Quantization still helps with throughput (fewer bytes per weight fetch), but the "can't fit the model" pressure is gone. Quantize for throughput, not memory — FP8 on H100 is still worth it; INT4 on A100 may or may not be depending on your latency requirements and whether the 1-2% quality drop matters for your use case. The decision that trips most engineers up: they reach for quantization as a last resort, after the model is already deployed and a cost spike happens. The better approach is to decide format before deployment, using the hardware you have and the quality bar your application actually requires. Run a task-specific eval on a 100-sample golden set before and after quantization. If the gap is within your tolerance, ship it. If not, go one level higher in precision and re-evaluate. The [evaluation article](/ai-engineering/articles/why-llm-eval-is-hard) covers how to build that eval set quickly. One more thing: quantization is not a substitute for model selection. A well-quantized 70B model is better than a poorly-quantized one, but a 7B model at the right precision for your hardware will always outperform a 70B model running out-of-memory under swap. Size and format together determine your actual serving economics — neither decision is separable from the other. The throughput vs latency economics are covered in full in the [GPU economics article](/ai-engineering/articles/throughput-vs-latency-gpu-economics); the [serving engine comparison](/ai-engineering/articles/serving-engine-comparison-2026) covers how vLLM, SGLang, TensorRT-LLM, and Ollama each handle these formats differently. ## Frequently asked questions Q: What is LLM quantization and why does it matter? A: Quantization reduces the numerical precision of model weights from 16-bit floats (BF16/FP16) down to 8, 4, or even 2 bits. A 70B model in BF16 needs ~140 GB of VRAM; at INT4 that drops to ~35 GB, fitting on a single A100-80GB (with room left for KV cache) instead of four. The tradeoff is a small quality loss — typically 1-2% on MMLU/HumanEval at 4-bit — which is acceptable for most production tasks. Below 3 bits, accuracy degrades sharply. Q: What is the difference between GPTQ and AWQ? A: GPTQ minimizes per-layer reconstruction error using a second-order Hessian approximation; it requires calibration data and can take hours on large models. AWQ (Activation-aware Weight Quantization) instead identifies the ~1% of weights that have the largest activation magnitudes and protects them at higher precision, running in tens of minutes. AWQ delivers better quality on reasoning tasks in practice and has become the default for new 4-bit deployments as of 2026. Q: When should I use GGUF instead of AWQ or GPTQ? A: GGUF (the llama.cpp format) is your only practical choice when running on CPU, Apple Silicon, or hybrid CPU+GPU setups. It supports a range of sub-formats from Q2_K to Q8_0 and is the native format for Ollama. On a pure NVIDIA GPU stack in a production multi-user server, AWQ or FP8 will outperform GGUF in throughput because GPU-optimized CUDA kernels are more efficient than llama.cpp kernels. Q: What is FP8 quantization and which GPUs support it? A: FP8 (8-bit floating point, specifically the E4M3 or E5M2 variants) is hardware-native on NVIDIA H100, H200, B200, Ada Lovelace GPUs (RTX 4090, L4, L40S), and the RTX 5090. Unlike INT4 which uses fixed-point arithmetic, FP8 retains a floating-point exponent, which means its dynamic range is much closer to BF16. Quality loss is near-zero compared to BF16. At moderate batch sizes on H100 SXM, FP8 inference runs roughly 1.8× faster than BF16 for 70B-class models — because halving bytes per weight directly increases effective HBM throughput. FP8 is the datacenter default for 2026 on H100+ hardware. Q: How much quality do you actually lose at 4-bit quantization? A: On standard benchmarks like MMLU and HumanEval, well-implemented 4-bit quantization (AWQ or GPTQ) shows roughly 1-2% absolute accuracy drop compared to BF16. For factual Q&A and coding this is usually invisible. For long multi-hop reasoning chains or math olympiad problems, the error accumulates more noticeably. Below 3 bits (Q2_K in GGUF, for instance) expect 5-15% degradation — the model starts confusing basic facts. Q: Can I quantize a model myself, or should I use a pre-quantized version? A: Both approaches work. Hugging Face Hub hosts pre-quantized versions for most popular models in GPTQ and AWQ formats; these are the fastest path to production. For custom fine-tuned models or when you need control over calibration data, run AWQ yourself with the autoawq library (tens of minutes on a single A100) or GPTQ with AutoGPTQ (hours, but more configuration options). Producing an FP8 checkpoint is just numeric conversion plus scale calibration and runs on any GPU — only the inference-time throughput benefit requires FP8-native hardware (Hopper, Blackwell, or Ada).