Quantization Demystified: GPTQ, AWQ, GGUF, and FP8 With Real Tradeoffs

A 70B model in BF16 costs you $14.40/hour on a four-A100 node on AWS (four A100-80GB at $3.60/GPU/hour, illustrative mid-2026 pricing). Run that continuously and you're at ~$10,000/month before you've processed a single user request. Cut to a single GPU with INT4 quantization and the monthly bill drops roughly 4× — while the model still passes most evals within 2 percentage points of its original score. That is the core premise of quantization, and it is real. The complication is that "INT4" is not one thing: GPTQ, AWQ, GGUF, and FP8 each make different tradeoffs, require different hardware, and produce different throughput characteristics. Picking the wrong one for your stack doesn't just cost money — it can produce a model that looks fine on a benchmark and quietly regresses on the one task your users actually care about.

What quantization is actually doing

Floating-point weights take 16 bits each in BF16. Quantization maps them to a smaller set of discrete values — 256 values at INT8, 16 values at INT4. The challenge is that not all weights are equal: a handful of them have large activation magnitudes and disproportionately influence the output. Compress those naively and you get visible quality loss. Protect them and you can push everything else to 4 bits without the model noticing much.

The three mechanisms that matter:

Per-group scaling. Instead of one scale factor per tensor (too coarse), use one scale per group of 128 weights. This means a 4-bit weight is decoded as fp16_value = int4_weight × per_group_scale + per_group_zero_point. The scales are stored in FP16, adding roughly 12% overhead over pure 4-bit storage, but group sizes of 64 or 128 are the sweet spot where quality holds.

Mixed-precision weight protection. Some formats identify individual outlier weights and store them at higher precision (FP16 or FP8) alongside the quantized weights. This is computationally more expensive but matters for models where activation outliers are concentrated in specific attention heads.

Hardware arithmetic. On H100/H200/B200, the tensor cores natively execute FP8 matrix multiplications at full hardware speed. INT4 kernels on older NVIDIA GPUs must unpack weights at runtime — still fast, but not hardware-native.

GPTQ: the original, still everywhere

GPTQ (2022, Frantar et al.) is the oldest widely-used post-training quantization method for LLMs. It processes each layer sequentially and minimizes the layer-wise reconstruction error using second-order Hessian information — essentially finding the INT4 values that make the quantized layer's output match the original layer's output as closely as possible on a calibration dataset.

flowchart TD
    W["Full-precision weight matrix W\n(BF16)"] --> CAL["Calibration forward pass\n(128–512 calibration samples)"]
    CAL --> H["Compute Hessian H\n(second-order info about sensitivity)"]
    H --> OPT["Layer-wise quantization\n(solve: min ||WX - Ŵ_q X||² )"]
    OPT --> QW["Quantized weights Ŵ_q\n(INT4 + FP16 group scales)"]
    style W fill:#0e7490,color:#fff
    style QW fill:#00e5ff,color:#0a0a0f
    style H fill:#a855f7,color:#fff

The practical results: GPTQ at 4-bit with group size 128 achieves perplexity within 0.1–0.5 points of the BF16 original on standard language modeling benchmarks. On task-specific evals like MMLU the gap is typically 1-3 percentage points. The quality is good — not great for complex reasoning, adequate for most production use cases.

The cost is time and calibration data. Quantizing a 70B model with GPTQ takes roughly 4-8 hours on a single A100. The calibration dataset matters: using C4 (the default) works well for general tasks, but if your workload is code generation, using code-heavy calibration data produces meaningfully better results.

GPTQ is the dominant format on Hugging Face Hub because it has been around the longest. If you need a quantized model for a recently released open-weight model, GPTQ is usually available within days of the BF16 release, produced by community members using tools like AutoGPTQ. vLLM and SGLang both support GPTQ natively.

The throughput story for GPTQ is that weights are stored as INT4 but the GPU executes matrix multiplications after dequantizing to FP16 on the fly. This makes GPTQ memory-bandwidth-efficient (loading 4-bit weights takes half the time of BF16) but not compute-efficient (the actual multiply-add is FP16). For decode-phase inference — which is always memory-bandwidth-bound, as the prefill vs decode article explains — this is exactly the bottleneck you want to reduce.

AWQ: better quality, faster to produce

AWQ (Activation-aware Weight Quantization) takes a different angle. Rather than minimizing per-layer reconstruction error, it observes which weights have large corresponding input activations and protects those weights — keeping them at FP16 or scaling them more carefully before quantization.

The insight: LLM weights have activation outliers concentrated in a small fraction of channels (~0.1-1%). If you quantize those channels to INT4 with the same group scaling as everything else, you lose disproportionate quality. AWQ identifies these salient weights by measuring activation magnitudes on a calibration set, then scales the entire weight channel down by a per-channel factor before quantization and compensates by scaling activations up — effectively applying a smoother quantization to the sensitive channels.

# Illustrative AWQ quantization with autoawq (not full production config)
from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_path = "meta-llama/Meta-Llama-3.1-70B-Instruct"
quant_path = "llama-3.1-70b-awq-int4"

model = AutoAWQForCausalLM.from_pretrained(model_path, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

quant_config = {
    "zero_point": True,
    "q_group_size": 128,  # group size for per-group scaling
    "w_bit": 4,           # 4-bit weights
    "version": "GEMM",    # GEMM kernel variant (vs GEMV for small batch)
}

# Calibration data — domain matters; use samples representative of your workload
model.quantize(tokenizer, quant_config=quant_config)
model.save_quantized(quant_path)
tokenizer.save_pretrained(quant_path)

The practical advantage over GPTQ: AWQ runs in tens of minutes rather than hours, and the quality on reasoning-heavy benchmarks is consistently better. A 2024 comparison of AWQ vs GPTQ on Llama-2 70B showed AWQ with 0.3-0.5 percentage points higher MMLU than GPTQ at the same bit-width, with a larger gap on GSM8K (math reasoning). As of 2026, AWQ is the recommended default for new 4-bit production deployments.

Both vLLM and SGLang load AWQ models directly:

# vLLM serving with AWQ
from vllm import LLM, SamplingParams

llm = LLM(
    model="llama-3.1-70b-awq-int4",
    quantization="awq",
    gpu_memory_utilization=0.85,  # leave headroom for KV cache
    max_model_len=8192,
)
params = SamplingParams(temperature=0.7, max_tokens=512)
outputs = llm.generate(["Explain quantization to an engineer."], params)

One gotcha: AWQ GEMM kernels are optimized for batch sizes ≥ 8. For single-request or very small batch inference (a chatbot with one user), the GEMV variant is faster. vLLM selects this automatically, but if you're calling kernels directly, pick the right one.

GGUF: the CPU-and-local-inference format

GGUF is the file format used by llama.cpp, which supports a range of quantization levels under a unified naming scheme: Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and a few others. The letter suffix (K, M, S) refers to variant strategies within each bit-width. Q4_K_M is the practical default — 4-bit with medium-quality K-quant scaling.

flowchart LR
    Q2["Q2_K\n~26 GB for 70B\nStrong degradation\nSurvival mode only"]
    Q4["Q4_K_M\n~40 GB for 70B\n2-3% quality drop\nOllama default"]
    Q6["Q6_K\n~58 GB for 70B\n<1% quality drop\nCPU quality sweetspot"]
    Q8["Q8_0\n~70 GB for 70B\n~0% quality drop\nPractically lossless"]
    Q2 --> Q4 --> Q6 --> Q8
    style Q2 fill:#ff2e88,color:#fff
    style Q4 fill:#ffaa00,color:#0a0a0f
    style Q6 fill:#15803d,color:#fff
    style Q8 fill:#0e7490,color:#fff

GGUF's primary advantage is hardware flexibility. llama.cpp can run on CPU, Apple Silicon (using Metal), NVIDIA GPUs, AMD GPUs, and mixed CPU+GPU setups where part of the model offloads to system RAM. For a developer running a 7B model on an M3 MacBook Pro or a team deploying to edge hardware without a GPU, GGUF is the only practical option.

Ollama uses GGUF as its native format. Pull a model with ollama pull llama3.1:70b and you're getting the Q4_K_M GGUF variant.

The tradeoff on pure NVIDIA GPU serving: GGUF kernels in llama.cpp are slower than GPTQ or AWQ kernels in vLLM for multi-user batch workloads. The gap is meaningful — roughly 20-30% lower throughput at batch sizes above 8 on identical hardware. llama.cpp does implement continuous batching (and Ollama exposes parallel request slots), but the scheduler, memory management, and high-batch kernels are far less efficient than vLLM's PagedAttention stack. Running GGUF/Ollama as your production inference server for multiple concurrent users means leaving a large fraction of your GPU's throughput on the table. For local dev or single-user internal tools, this is fine. For a multi-user product, it is not.

{ "type": "kv-cache", "model": "70b", "context": 4096, "batch": 16, "gpu": "a100-80", "title": "VRAM breakdown: 70B model at BF16 (weights + KV cache at 16 concurrent requests)" }

FP8: the datacenter default for H100+ hardware

FP8 is categorically different from the INT4 approaches above. It uses 8-bit floating-point representation — for inference, both weights and activations use NVIDIA's E4M3 variant; the wider-exponent E5M2 is reserved for training gradients (and shows up as an optional KV-cache dtype) — which retains a floating-point exponent. The dynamic range is close to BF16, which means quantization error is dramatically smaller than INT4.

NVIDIA H100, H200, B200, the Ada Lovelace generation (RTX 4090, L4, L40S), and the RTX 5090 have hardware tensor cores that execute FP8 matrix multiplications natively. This is not a software emulation — the arithmetic runs at full hardware speed, and the throughput gain comes from loading half the bytes from HBM per multiply-accumulate. On H100 SXM5, FP8 inference achieves roughly 1.8× the BF16 throughput on decode-bound workloads.

Quality-wise, FP8 is almost indistinguishable from BF16. Published benchmarks on Llama 3.1 70B show less than 0.1 percentage point MMLU difference. For tasks that are already within BF16's ability, FP8 is essentially lossless. This makes it the natural first choice on datacenter hardware.

# vLLM with FP8 (H100/H200 required)
from vllm import LLM

llm = LLM(
    model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    quantization="fp8",          # hardware-native on H100/H200
    kv_cache_dtype="fp8",        # also quantize KV cache to FP8
    gpu_memory_utilization=0.90,
)
# FP8 KV cache halves KV memory, doubling concurrent request capacity
# See /articles/kv-cache-gpu-memory-math for the full calculation

The kv_cache_dtype="fp8" line above is worth pausing on. The KV cache article covers the memory formula in detail, but the short version: FP8 KV cache on a 70B model at 32 concurrent × 8K context saves ~42 GB compared to BF16 KV cache (0.32 MB/token × 32 × 8192 ≈ 84 GB, halved). That is often more impactful than the weight compression itself at moderate context lengths.

FP8 quantization of weights is fast — models like Llama 3 70B quantize in minutes using NVIDIA's modelopt toolkit or vLLM's built-in quantization — and the calibration dataset matters less than for INT4 because the quantization error is so small to begin with.

The constraint: native FP8 compute needs FP8 tensor cores — Hopper (H100/H200), Blackwell, or Ada (RTX 4090, L4, L40S). On Ampere (A100 and older), vLLM falls back to weight-only FP8 via Marlin kernels: you keep the halved weight memory and bandwidth (often still faster than BF16 on memory-bound decode), but compute upconverts to FP16 and you get none of the FP8 tensor-core throughput. If your stack is A100 or older and you want real compression wins, AWQ INT4 is the stronger call.

The quantization decision tree

flowchart TD
    START["What hardware?"] --> H100["FP8-native: H100 / H200 / B200\nAda (4090, L4, L40S) / RTX 5090"]
    START --> A100["A100 / A10G / older NVIDIA"]
    START --> CPU["CPU or Apple Silicon\nor hybrid CPU+GPU"]
    H100 --> FP8_REC["Use FP8\nnear-lossless, fastest throughput\nvLLM fp8 or modelopt"]
    A100 --> MULTI{"Multi-user\nproduction server?"}
    MULTI -->|"Yes"| AWQ_REC["Use AWQ INT4\nbest quality-per-bit on CUDA\nautoawq + vLLM/SGLang"]
    MULTI -->|"No (local dev / single user)"| GGUF_OK["GGUF Q4_K_M is fine\nOllama or llama.cpp"]
    CPU --> GGUF_REC["Use GGUF Q4_K_M\n(or Q6_K for higher quality)\nlllama.cpp / Ollama"]
    style FP8_REC fill:#a855f7,color:#fff
    style AWQ_REC fill:#00e5ff,color:#0a0a0f
    style GGUF_REC fill:#ffaa00,color:#0a0a0f
    style GGUF_OK fill:#15803d,color:#fff

GPTQ doesn't appear prominently in this tree because in 2026 AWQ is strictly better for new deployments: better quality, faster to produce, equally supported. GPTQ remains relevant because it dominates the existing catalog of quantized models on Hugging Face Hub — if a model you need is only available as GPTQ, use it. If you're producing the quantization yourself, use AWQ.

Worked numbers: serving cost at three quantization levels

Suppose you're pricing a single serving replica for a 70B model (requests averaging 1K input + 500 output tokens). One replica at these throughputs handles on the order of 100-190 requests/minute; scale the fleet linearly for more — the per-token economics below are what carry.

Hardware: NVIDIA A100-80GB, $3.60/GPU/hour (illustrative, mid-2026 on-demand AWS)

BF16 baseline:
  Weights: 140 GB → 2 A100s (with 20 GB left for KV cache — tight)
  More realistic: 4 A100s for any reasonable concurrency
  At 4 GPUs: $14.40/hour = ~$10,400/month
  Throughput: ~800 tok/s decode on 4 A100s (memory-bandwidth-bound)

AWQ INT4:
  Weights: ~38 GB → fits on 1 A100 with 42 GB KV cache headroom
  At 1 GPU: $3.60/hour = ~$2,600/month
  Throughput: ~900 tok/s decode on 1 A100 (less weight bytes to load per step)
  Cost per million output tokens: $3.60 / (900 × 3600 / 1M) = ~$1.11/M tok

FP8 on H100-80GB ($8.00/hour illustrative):
  Weights: ~72 GB → 1 H100 with 8 GB remaining
  Better: 1 H100 with FP8 KV cache for 50+ concurrent requests
  At 1 H100: $8.00/hour = ~$5,760/month
  Throughput: ~1,600+ tok/s decode (H100 HBM bandwidth + FP8 tensor cores)
  Cost per million output tokens: $8.00 / (1600 × 3600 / 1M) = ~$1.39/M tok

Summary:
  AWQ on A100 = 4× cost reduction vs BF16 baseline (on same A100 hardware)
  FP8 on H100 = 1.8× throughput vs BF16 on same H100, ~45% lower cost vs H100+BF16

The conclusion from this math: if you're on A100 hardware, AWQ INT4 is the single highest-impact change you can make to inference cost. If you're upgrading to H100 anyway, FP8 is the obvious format — the quality improvement over AWQ justifies the format change even if the cost numbers come out similar.

What breaks

Quality collapse below 3 bits. The Q2_K GGUF and similar 2-bit formats sound appealing on paper (a 70B model in ~26 GB — K-quants store scales, so "2-bit" really means ~2.6-3.4 bits per weight). In practice, 2-bit quantization destroys enough numerical precision that the model confuses basic facts, loses coherence on long outputs, and fails structured tasks. The perplexity numbers look acceptable on benchmarks but real task performance falls off a cliff. Stay at 4-bit as your floor for anything user-facing.

Calibration data mismatch. GPTQ's Hessian approximation and AWQ's salient weight detection both depend on calibration data. Using generic Wikipedia/C4 calibration for a code-generation model means the quantization wasn't optimized for your actual distribution. The fix: use 128-512 samples from your target domain. For a coding assistant, that means code. The quality difference between matched and mismatched calibration at 4-bit can be 2-4 percentage points on task-specific evals.

GGUF in production GPU servers. Running Ollama (GGUF-backed) as a multi-user production inference server is the most common mistake teams make when transitioning from local prototyping to deployment. Ollama does batch concurrent requests (via llama.cpp's parallel slots), but its scheduling and memory management are far less efficient at high concurrency than vLLM's continuous batching with PagedAttention. The throughput difference is not 10% — it is often 3-10× at 50+ concurrent users. See the continuous batching article for the mechanics.

Kernel availability for new models. The throughput gains from quantization depend on efficient GPU kernels, not just the reduced memory footprint. When a new model architecture ships, the AWQ/GPTQ kernels in vLLM take a few weeks to be validated. In that window, quantized inference may fall back to unoptimized paths and actually be slower than BF16. Check the vLLM model support table before assuming quantization will speed things up on a brand-new architecture.

FP8 on non-FP8-native hardware. If you run quantization="fp8" in vLLM on an A100, it falls back to weight-only FP8 (Marlin kernels) with a logged warning — you keep the memory savings and often some decode speedup, but none of the native FP8 compute throughput the H100 benchmarks advertise. Miss the warning and you'll conclude FP8 is broken when it's actually running in a degraded mode. Know which mode your hardware gets before you benchmark.

KV cache precision mismatch. Quantizing weights to INT4 while leaving the KV cache at BF16 leaves the bigger memory consumer untouched at high concurrency. Conversely, quantizing the KV cache to FP8 when your weights are still BF16 on an H100 is often the higher-value move — KV cache can exceed weight memory at moderate context lengths and batch sizes. Consider weight quantization and KV cache quantization independently. (The interactive widget above makes this concrete for a 70B model.)

When to use what in practice

New deployment, H100/H200/B200: Start with FP8. Run vLLM with quantization="fp8" and kv_cache_dtype="fp8". Quality is near-BF16, throughput is the highest available. Only reach for INT4 if you need to fit a larger model onto the same GPU count.

Existing A100 or A10G fleet: AWQ INT4 is the call. Produce the quantized model with autoawq, push to your model registry, serve with vLLM or SGLang. Expect ~2× weight-loading speedup on decode-bound inference and a significant reduction in GPU count required.

Local development or internal tooling with a single user: GGUF with Ollama. It is the fastest path from model name to running inference and requires zero serving infrastructure. Use Q4_K_M for the quality/size balance, Q6_K if you have the VRAM and care about accuracy. Don't run it for anything requiring concurrent requests at scale.

Existing GPTQ model from Hugging Face, no re-quantization budget: Use it. GPTQ is supported in vLLM and SGLang and the quality is good enough for most tasks. If quality matters and you have time, re-quantize to AWQ — the improvement on reasoning tasks is real. If you don't have time, ship the GPTQ model.

Model smaller than 13B: At 7-8B parameters, BF16 fits in a single A100-80GB with ample KV cache headroom. Quantization still helps with throughput (fewer bytes per weight fetch), but the "can't fit the model" pressure is gone. Quantize for throughput, not memory — FP8 on H100 is still worth it; INT4 on A100 may or may not be depending on your latency requirements and whether the 1-2% quality drop matters for your use case.

The decision that trips most engineers up: they reach for quantization as a last resort, after the model is already deployed and a cost spike happens. The better approach is to decide format before deployment, using the hardware you have and the quality bar your application actually requires. Run a task-specific eval on a 100-sample golden set before and after quantization. If the gap is within your tolerance, ship it. If not, go one level higher in precision and re-evaluate. The evaluation article covers how to build that eval set quickly.

One more thing: quantization is not a substitute for model selection. A well-quantized 70B model is better than a poorly-quantized one, but a 7B model at the right precision for your hardware will always outperform a 70B model running out-of-memory under swap. Size and format together determine your actual serving economics — neither decision is separable from the other. The throughput vs latency economics are covered in full in the GPU economics article; the serving engine comparison covers how vLLM, SGLang, TensorRT-LLM, and Ollama each handle these formats differently.