Decoding and Sampling: Temperature, Top-P, Min-P, and When to Use Each

A customer files a bug: your document summarizer occasionally returns a summary that includes a sentence about a competitor product that appears nowhere in the source document. You check the prompt — clean. You check the retrieval — the right document came back. You check the model — same version as always. The culprit is temperature=0.9, top_p=1.0 shipping in production on a task that needs nearly-deterministic output. Every thousand summaries, a low-probability token slips through and the model confabulates a sentence around it.

That is the practical consequence of mismatched decoding parameters. The model had the right information. The sampling configuration handed control to the random tail of the distribution.

What actually happens at decoding time

The transformer's forward pass produces one vector: the logits, a float32 array with one score per vocabulary token. For a 32K-vocabulary model that is 32,000 numbers. Nothing in the forward pass constrained which tokens appear — the entire vocabulary is in play.

Decoding is the series of transformations that turn that logit vector into a single selected token. You repeat this process once per output token until the model emits the end-of-sequence token or you hit a configured limit.

sequenceDiagram
    participant M as Model forward pass
    participant T as Temperature scaling
    participant S as Softmax
    participant F as Vocabulary filter (K/P/min-p)
    participant D as Draw token

    M->>T: logit[0..V] (raw scores)
    T->>S: logit[i] / temperature
    S->>F: probability[0..V] (sums to 1.0)
    F->>D: pruned, renormalized distribution
    D->>M: selected token → next input

Each step in that pipeline has a parameter. Understanding what each one does — and in what order they apply — is the entire article.

Temperature: the only knob that touches the logits

Before softmax converts logits into probabilities, each logit is divided by a scalar T. That's it. Temperature does not remove any tokens from consideration; it changes how spread out the resulting probability distribution is.

When T < 1.0, dividing stretches the gaps between logits — a gap of 8 between the top two logits becomes a gap of 16 at T=0.5. Softmax weights each token by e raised to its logit, so probability ratios depend on those gaps: widen the gap and the leader's share grows exponentially. The result is a sharper distribution: the highest-probability token takes an even larger share, and low-probability tokens shrink toward zero.

When T > 1.0, dividing compresses the differences. A logit of 10 and a logit of 2 become 10/1.5 ≈ 6.7 and 2/1.5 ≈ 1.3 — closer together. Softmax produces a flatter distribution, and tokens that were unlikely get a real chance at selection.

{
  "type": "sampler",
  "prompt": "story-open",
  "temperature": 0.7,
  "title": "How temperature, top-p, and min-p reshape the token distribution"
}

T=0 is a degenerate case. Division by zero is avoided in practice by treating it as T → 0, which collapses the distribution to a point mass at the argmax — always pick the single highest-scoring token. This is greedy decoding. It is not truly deterministic: hardware non-determinism (floating-point accumulation order varies across CUDA cores), different batch sizes, and different GPU architectures can produce different results even at T=0. If you need reproducibility, you need a fixed random seed and consistent hardware and batch size of one — which is often impractical.

Practical guidance: temperature is the first thing to get right because it sets the ceiling on randomness. Subsequent filters can only remove tokens from contention; they cannot add tokens back or change the fundamental shape that temperature established. One caveat: this temperature-then-filter order is how HF transformers and most hosted APIs describe their pipelines, but llama.cpp's default sampler chain applies temperature after the filters — which is part of why min-p there keeps output coherent even at temperatures well above 1.

Top-K: the bluntest filter

Top-K keeps exactly the K highest-probability tokens and throws out the rest. If K=50, only 50 tokens can ever appear as the next output, regardless of how flat or peaked the distribution is.

The problem is that K is context-free. When the model is highly confident — say, 92% probability on the token "Paris" after "The capital of France is" — top-K=50 still keeps 49 other candidates that the distribution has already assigned near-zero probability. When the model is genuinely uncertain — choosing the next word of a creative passage where many continuations are plausible — top-K=50 might be too restrictive.

Top-K works fine as a rough sanity cap (top-K=1 is greedy decoding, top-K=200,000 is unconstrained sampling), but in practice it has been largely supplanted by top-p and min-p for anything requiring nuance. Most production configurations still set a top-K as a hard ceiling (e.g., top-K=100) and rely on top-p or min-p to do the fine-grained work.

Top-P: adaptive but imperfect

Top-P, introduced as "nucleus sampling" in The Curious Case of Neural Text Degeneration (2020), takes a different approach: instead of a fixed number of candidates, keep the smallest set of tokens whose cumulative probability reaches P.

If top_p=0.9, sort tokens by probability descending, accumulate until you hit 90% of total probability mass, keep that set, discard the rest. When the model is confident (peaked distribution), you might keep 3 tokens to reach 90%. When the model is uncertain (flat distribution), you might need 200 tokens to reach 90%.

This is more adaptive than top-K. But it has a subtle failure mode: on a genuinely flat distribution, the tail tokens that squeeze in just before the 90% threshold have very low individual probabilities — say, 0.1% each. They are statistically marginal but syntactically wrong. Top-P includes them anyway once the bulk of the probability mass is distributed across many tokens.

Another common mistake: top_p=1.0. This includes the entire vocabulary. People set it thinking they are "disabling" top-P filtering, but all they have done is removed the filter entirely. With a moderate temperature, this means genuinely low-probability garbage tokens can appear. The correct way to disable top-P filtering is to not use it, or to use top_p=1.0 only when paired with a near-zero temperature.

Min-P: the 2024 upgrade that open-source runtimes adopted

Min-P was proposed in 2024 and within a year became the default sampling method in llama.cpp, Ollama, and several other open-source runtimes, displacing top-P as the primary filter.

The idea is to make the cutoff proportional to the top token's probability rather than to a fixed cumulative mass. Specifically: compute the top token's probability p_max. Set the minimum threshold at min_p × p_max. Discard any token whose probability is below this threshold.

When the model is confident (p_max = 0.80), and min_p=0.05, the cutoff is 0.05 × 0.80 = 0.04. Any token below 4% probability is dropped — a reasonably strict filter.

When the model is genuinely uncertain (p_max = 0.08), the cutoff drops to 0.05 × 0.08 = 0.004. Many tokens survive, reflecting the real ambiguity of the situation.

flowchart TD
    PM["Top token probability p_max"] --> CALC["Threshold = min_p × p_max"]
    CALC --> FILTER["Keep tokens where p_i ≥ threshold"]
    FILTER --> RENORM["Renormalize remaining probabilities"]
    RENORM --> SAMPLE["Sample from filtered set"]

    style PM fill:#a855f7,color:#fff
    style CALC fill:#ffaa00,color:#0a0a0f
    style SAMPLE fill:#00e5ff,color:#0a0a0f

The result: min-P automatically adapts its strictness to the model's confidence at each decoding step. Top-P has a fixed target (90% of the probability mass); min-P has a relative target (be at least 5% as likely as the most likely token).

In practice, min-P reduces nonsensical insertions on creative tasks — the kind where a story about a detective suddenly produces "banana" because top-P included low-probability tokens that happened to sit just inside the 95% cumulative window. Min-P discards those because they are nowhere near the probability of the next natural narrative token.

A sensible starting point for creative work: temperature=0.8, min_p=0.05. For structured outputs: temperature=0.1, min_p=0.05 (or just temperature=0, ignoring min-P entirely since greedy already takes the argmax).

Parameter comparison

Method	What it cuts	Adapts to distribution?	Main failure mode
Greedy (T=0)	Everything except argmax	N/A	Repetitive, mode-collapsed output
Top-K	All but top K tokens	No — fixed count	Too strict when model confident; too loose when uncertain
Top-P	Tokens outside cumulative P mass	Partially — variable count	Includes low-p tail on flat distributions; p=1.0 is a footgun
Min-P	Tokens below min_p × p_max	Yes — threshold scales with p_max	Setting min-p too high on uncertain distributions can over-prune
Beam search	Non-beam paths	N/A (deterministic)	Computationally expensive; produces generic/averaged outputs

Beam search deserves a brief mention since it appears in older documentation: it maintains B candidate sequences in parallel at each step and keeps the top-B by cumulative log-probability. It is deterministic and tends to produce higher-likelihood but lower-diversity text. Modern practice has almost entirely moved away from it for generative tasks — the computational cost is B× the single-sequence cost and the outputs are worse for open-ended generation. It still shows up in constrained settings like machine translation with strict fluency requirements.

The numbers: what these parameters cost you

None of the sampling parameters add meaningful compute to the forward pass. The logit vector exists after the forward pass; filtering and sampling are O(V log V) sort operations on CPU or a simple parallel reduction on GPU — microseconds at most. The only sampling technique that has real latency implications is speculative decoding, and that is a reduction, not an increase.

Back-of-the-envelope: sampling overhead per token

Forward pass for a 70B model at batch=1, A100 80GB:
  ~15-25ms per token (decode phase, memory-bandwidth bound)

Sorting 128K logits for top-K filter:
  ~0.02ms (trivially parallelizable on GPU)

Top-P or Min-P filter (linear scan after sort):
  ~0.001ms

Total sampling overhead as fraction of forward pass:
  < 0.2%

Conclusion: sampling parameters do not affect latency.
The real latency lever is output token count, not sampling method.

The token count is affected by sampling choice in a subtle way: greedy decoding can fall into repetition loops that inflate output length. Filters like min-P and top-P break these loops by preventing the model from always returning to the same high-probability sequence. Setting a repetition_penalty (multiplicatively discounting already-generated tokens in the logit vector before softmax) is a complementary mechanism — not a sampling parameter per se, but usually mentioned alongside them.

Reasoning models: where the API lies to you

OpenAI's o1, o3, o4-mini, and Google's Gemini 2.0/2.5 Flash Thinking operate differently. These models generate extended internal reasoning traces before producing output. The sampling inside that reasoning chain is managed by the model and is not exposed through the API.

Some providers accept a temperature parameter for compatibility but clamp it to a narrow internal range. Others return an error if you attempt to set it. A few silently ignore it. The model's output is not actually drawn from the distribution you think you are shaping.

For these models, the useful parameters to tune are:

• max_tokens / max_completion_tokens: controls how long the reasoning chain can run, which affects quality on hard problems
• reasoning_effort (OpenAI o-series): controls thinking budget explicitly on some models
• Top-P, temperature: treat as no-ops unless the provider documentation explicitly says otherwise

This matters for cost calculations too. The internal reasoning tokens count toward your bill even though they are not returned in the response body (depending on provider). See token economics for the billing mechanics.

What breaks

Greedy repetition loops

The most common failure of greedy decoding on open-ended text: the model produces a sentence, and by the time it finishes the sentence, the most likely next token is the first word of the same sentence again. You get output like: "The solution is to check the configuration file. The solution is to check the configuration file. The solution is to check the configuration file." indefinitely.

Greedy does not break these loops because the argmax is always the most probable token — and for a repetitive prefix, the most probable next token is more repetition. Adding a small temperature (0.2-0.4) or a repetition penalty breaks the cycle.

Top-P undershooting on peaked distributions

If top_p=0.95 and the model assigns 97% probability to a single token, top-P keeps only that token (or maybe two). That sounds like greedy decoding, but it is not — you still sampled it. The issue is that on every other step where the model is uncertain, top-P may include problematic tail tokens. The parameter cannot be tuned to work well in both regimes simultaneously.

Min-P passes (almost) everything on flat distributions

With min_p=0.1 and a very flat distribution where no token exceeds 2% probability, the threshold becomes 0.1 × 0.02 = 0.002. Almost nothing gets pruned. This is correct behavior — min-P correctly diagnoses "the model is deeply uncertain; keep many candidates." But if your application needs some filter in this case (say, you need to avoid certain token classes), min-P alone is insufficient and you need an additional vocabulary constraint like logit biases or constrained decoding. The opposite failure exists too: set min_p too high and you over-prune — at min_p=0.5 with p_max=0.08, only tokens above 4% survive, which on a genuinely flat distribution can leave one or two candidates and collapse the diversity you were sampling for. See structured outputs and constrained decoding for the production approach.

Temperature=0 is not deterministic across hardware

Multiple engineers have been surprised to discover that the same prompt with temperature=0 produces different results on an A100 vs an H100, or even on the same GPU with a different batch size. The reason is floating-point non-associativity: changing the order of reduction operations in a matrix multiply changes the rounding errors, which can tip a close logit competition. The model you use in development (e.g., a hosted API) may use different hardware than your production inference server, producing different "greedy" outputs.

If you need reproducible outputs, you need: fixed hardware type, fixed batch size of one, and a fixed random seed even with temperature=0. Providers rarely guarantee this across deployments.

The creativity-factuality coupling

High temperature increases creativity and increases hallucination simultaneously. The same mechanism that makes the model produce novel phrasings also makes it draw from lower-probability, less-grounded token sequences. There is no temperature setting that maximizes creativity while minimizing hallucination — they are directly coupled through the distribution geometry.

The practical solution is to separate the tasks: use low temperature for factual extraction and grounding, use higher temperature for generation over retrieved facts. The why models hallucinate article covers the mechanistic reasons for this trade-off in detail.

Speculative decoding: separate from but worth knowing alongside

Speculative decoding is not a sampling parameter — it is an inference optimization that preserves your sampling distribution exactly. A small draft model (typically 1-7B parameters) generates K tokens autoregressively, then the target model (the one you actually want) verifies all K tokens in a single parallel forward pass. Each token is accepted or rejected based on whether the target model's distribution agrees. Accepted tokens carry through; the first rejected token is resampled from the corrected distribution.

The math guarantees that the resulting token sequence is drawn from exactly the same distribution as if the target model had generated every token independently. Two to four times speedup at zero quality change — the settings you configure for temperature, top-p, and min-p on the target model still apply, and the output statistics are identical. See speculative decoding for the mechanics.

The decision in practice

Pick your sampling configuration based on what kind of failure you cannot tolerate.

You cannot tolerate hallucination or format errors: use temperature=0 or temperature=0.1. Accept that outputs will be repetitive on some long generations; add repetition_penalty=1.1 if that becomes a problem. Know where that knob exists, though: repetition_penalty is an open-source-runtime parameter (HF transformers, vLLM, llama.cpp) — OpenAI's closest equivalents are the additive frequency_penalty/presence_penalty, and Anthropic exposes neither, so on Claude the fix is a slightly higher temperature or a prompt-level instruction. For structured outputs, combine with constrained decoding — do not rely on temperature alone to enforce format.

You need diversity but care about coherence: temperature=0.7, min_p=0.05. This is a reliable default for most chat and Q&A applications. The min-P filter handles distribution spikes better than top-P, and temperature=0.7 gives enough variance that responses do not feel canned.

You need creative range: temperature=0.9-1.1, min_p=0.03. At this temperature range you will occasionally get odd outputs — that is the point. Run multiple samples and let your application logic or a downstream classifier select the best one. Best-of-N sampling at temperature=0.8 frequently outperforms a single sample at temperature=0 on creative tasks and on harder reasoning problems where the model benefits from exploring multiple paths.

You are calling a reasoning model: set temperature to whatever the documentation says and ignore the section above. Set your output token budget thoughtfully — that is the only real lever you have on quality.

from anthropic import Anthropic

client = Anthropic()

# Structured extraction — near-deterministic. One sampling knob only:
# the Anthropic API rejects requests that set both temperature and top_p —
# which is fine, because stacking a filter on a temp-0.1 distribution
# buys you nothing anyway.
extraction_response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=512,
    temperature=0.1,          # low, not zero — avoids hardware non-determinism issues
    messages=[{
        "role": "user",
        "content": "Extract the invoice total from the following text as JSON: ..."
    }]
)

# Creative generation — diversity wanted. Same rule: temperature OR top_p,
# never both. min_p is not exposed here at all — it lives in open-source
# runtimes like llama.cpp/Ollama.
creative_response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    temperature=0.9,
    messages=[{
        "role": "user",
        "content": "Write the opening paragraph of a noir detective story set in 2045."
    }]
)

For open-source runtimes with min-p support (Ollama, llama.cpp, vLLM with recent builds):

import openai  # vLLM exposes an OpenAI-compatible endpoint

client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="ignored")

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a short poem about debugging."}],
    temperature=0.8,
    extra_body={
        "min_p": 0.05,        # min-p passed as an extra field; not in OpenAI spec
        "repetition_penalty": 1.05,
    },
    max_tokens=256,
)

The extra_body pattern for min-p works because most OpenAI-compatible servers pass through unknown parameters to the underlying sampler. Check your specific server's documentation — parameter names vary (min_p, min_token_prob, min_probability).

If you are making decisions about which model to call for a given task — including whether to route to a reasoning model or a standard model — model routing and cascades covers the production pattern. And if you are debugging why outputs feel repetitive or hallucinated, reading why models hallucinate and tokenization and BPE side by side is worth the hour — the roots of many output quality problems trace to either the sampling configuration or the way the prompt tokenizes, not to the model's capabilities.