# Decoding and Sampling: Temperature, Top-P, Min-P, and When to Use Each > How temperature, top-p, top-k, and min-p reshape the next-token probability distribution — and which setting to reach for in production vs creative tasks. Canonical: https://ironclad.academy/ai-engineering/articles/decoding-sampling-temperature-top-p-min-p | Difficulty: medium | Topics: LLM Internals, Inference, Fundamentals ## Key takeaways - Temperature scales logits before softmax; in HF transformers and most hosted APIs the other sampling parameters only filter the candidate set after that scaling (llama.cpp's default chain applies temperature last). - Top-p=1.0 does not disable sampling; it includes the entire vocabulary and makes outputs noisier, not more faithful. - Min-p adapts its cutoff to the distribution shape at each step, producing fewer nonsensical insertions than top-p on creative tasks. - Temperature=0 gives greedy decoding, but hardware non-determinism means "greedy" is not perfectly reproducible across different hardware or batch sizes. - For reasoning models, temperature handling varies by provider — OpenAI o-series models reject or ignore it, Gemini thinking models accept it — but the internal reasoning sampling is never exposed, so check the docs before tuning. - Speculative decoding changes latency by 2-4x without touching output distribution — it is orthogonal to your sampling settings. > **In one line:** Decoding parameters are not knobs to fiddle randomly — each maps to a specific transformation of the next-token probability distribution that determines whether your model sounds like a textbook, a novelist, or a broken Markov chain. **The idea.** After the forward pass, the model has produced a logit vector with one score for every token in the vocabulary — 32,000 to 200,000 entries depending on the model. Decoding is the process of turning that vector into one sampled token, then repeating. Temperature, top-k, top-p, and min-p are applied in sequence, each reshaping or pruning the candidate set before the final draw. They do not change what the model *knows*; they change which part of what it knows shows up in the output. Getting this wrong is cheap to do and expensive to notice. Production teams routinely ship `temperature=1.0, top_p=1.0` on structured extraction tasks and then spend days debugging why JSON occasionally arrives with hallucinated fields. Or they set `temperature=0` on a creative writing feature and wonder why every story starts with "Once upon a time in a small village." **Key points.** - Temperature scales raw logits before softmax. It is applied first and affects everything downstream. - Top-k and top-p are vocabulary filters applied after temperature scaling. They prune the candidate set; the final sample comes from what remains. - Min-p (2024, now the default in llama.cpp and many open-source runtimes) adapts its threshold to the distribution shape at each step — strict when the model is confident, relaxed when uncertain. - Temperature=0 means greedy decoding — argmax, not sampling — but hardware non-determinism can still produce different results across different batch sizes or hardware. - For reasoning models, temperature is locked or irrelevant. The model's internal chain-of-thought sampling is not exposed. **The pipeline in one diagram.** ```mermaid flowchart LR A["Raw logits\n(vocab × float32)"] --> B["÷ Temperature\n(scales all logits)"] B --> C["Softmax\n→ probability distribution"] C --> D["Top-K filter\n(keep top K tokens)"] D --> E["Top-P / Min-P filter\n(prune low-probability tail)"] E --> F["Renormalize\n→ final distribution"] F --> G["Sample one token"] style A fill:#0e7490,color:#fff style G fill:#00e5ff,color:#0a0a0f ``` **Key design decisions.** | Task type | Temperature | Sampling strategy | Why | | --- | --- | --- | --- | | JSON/structured extraction | 0.0 – 0.2 | Greedy or top-p=0.9 | Predictability > diversity; schema errors are hard to catch | | Code generation (single best) | 0.0 – 0.3 | Greedy or top-k=40 | Syntactic correctness matters more than novelty | | Code generation (best-of-N) | 0.5 – 0.8 | top-p=0.95 or min-p=0.05 | Sample multiple candidates, pick the one that passes tests | | Chat / Q&A | 0.5 – 0.8 | top-p=0.9 or min-p=0.1 | Varied but coherent | | Creative writing | 0.8 – 1.1 | min-p=0.05 or top-p=0.95 | High diversity; min-p handles distribution spikes better than top-p | | Reasoning models | ignore | N/A | Provider controls internal sampling | **If you have 60 seconds, say this.** "The model produces a logit vector over the whole vocabulary. Temperature scales it — lower means sharper, higher means flatter. Then you apply a filter: top-k keeps the K biggest tokens, top-p keeps the smallest set that covers P of the cumulative probability, min-p drops anything below some fraction of the top token's probability. Whatever survives gets renormalized and you draw one token. For structured tasks use low or zero temperature. For creative tasks use moderate temperature plus min-p. For reasoning models, don't touch the temperature — the provider has already decided." A customer files a bug: your document summarizer occasionally returns a summary that includes a sentence about a competitor product that appears nowhere in the source document. You check the prompt — clean. You check the retrieval — the right document came back. You check the model — same version as always. The culprit is `temperature=0.9, top_p=1.0` shipping in production on a task that needs nearly-deterministic output. Every thousand summaries, a low-probability token slips through and the model confabulates a sentence around it. That is the practical consequence of mismatched decoding parameters. The model had the right information. The sampling configuration handed control to the random tail of the distribution. ## What actually happens at decoding time The transformer's forward pass produces one vector: the **logits**, a float32 array with one score per vocabulary token. For a 32K-vocabulary model that is 32,000 numbers. Nothing in the forward pass constrained which tokens appear — the entire vocabulary is in play. Decoding is the series of transformations that turn that logit vector into a single selected token. You repeat this process once per output token until the model emits the end-of-sequence token or you hit a configured limit. ```mermaid sequenceDiagram participant M as Model forward pass participant T as Temperature scaling participant S as Softmax participant F as Vocabulary filter (K/P/min-p) participant D as Draw token M->>T: logit[0..V] (raw scores) T->>S: logit[i] / temperature S->>F: probability[0..V] (sums to 1.0) F->>D: pruned, renormalized distribution D->>M: selected token → next input ``` Each step in that pipeline has a parameter. Understanding what each one does — and in what order they apply — is the entire article. ## Temperature: the only knob that touches the logits Before softmax converts logits into probabilities, each logit is divided by a scalar `T`. That's it. Temperature does not remove any tokens from consideration; it changes how spread out the resulting probability distribution is. When `T < 1.0`, dividing stretches the *gaps* between logits — a gap of 8 between the top two logits becomes a gap of 16 at `T=0.5`. Softmax weights each token by e raised to its logit, so probability ratios depend on those gaps: widen the gap and the leader's share grows exponentially. The result is a *sharper* distribution: the highest-probability token takes an even larger share, and low-probability tokens shrink toward zero. When `T > 1.0`, dividing compresses the differences. A logit of 10 and a logit of 2 become 10/1.5 ≈ 6.7 and 2/1.5 ≈ 1.3 — closer together. Softmax produces a *flatter* distribution, and tokens that were unlikely get a real chance at selection. [Interactive visualizer on the original page: sampler — How temperature, top-p, and min-p reshape the token distribution] `T=0` is a degenerate case. Division by zero is avoided in practice by treating it as `T → 0`, which collapses the distribution to a point mass at the argmax — always pick the single highest-scoring token. This is **greedy decoding**. It is not truly deterministic: hardware non-determinism (floating-point accumulation order varies across CUDA cores), different batch sizes, and different GPU architectures can produce different results even at `T=0`. If you need reproducibility, you need a fixed random seed *and* consistent hardware *and* batch size of one — which is often impractical. **Practical guidance:** temperature is the first thing to get right because it sets the ceiling on randomness. Subsequent filters can only remove tokens from contention; they cannot add tokens back or change the fundamental shape that temperature established. One caveat: this temperature-then-filter order is how HF transformers and most hosted APIs describe their pipelines, but llama.cpp's default sampler chain applies temperature *after* the filters — which is part of why min-p there keeps output coherent even at temperatures well above 1. ## Top-K: the bluntest filter Top-K keeps exactly the K highest-probability tokens and throws out the rest. If `K=50`, only 50 tokens can ever appear as the next output, regardless of how flat or peaked the distribution is. The problem is that K is context-free. When the model is highly confident — say, 92% probability on the token "Paris" after "The capital of France is" — top-K=50 still keeps 49 other candidates that the distribution has already assigned near-zero probability. When the model is genuinely uncertain — choosing the next word of a creative passage where many continuations are plausible — top-K=50 might be too restrictive. Top-K works fine as a rough sanity cap (top-K=1 is greedy decoding, top-K=200,000 is unconstrained sampling), but in practice it has been largely supplanted by top-p and min-p for anything requiring nuance. Most production configurations still set a top-K as a hard ceiling (e.g., top-K=100) and rely on top-p or min-p to do the fine-grained work. ## Top-P: adaptive but imperfect Top-P, introduced as "nucleus sampling" in [The Curious Case of Neural Text Degeneration (2020)](https://arxiv.org/abs/1904.09751), takes a different approach: instead of a fixed number of candidates, keep the *smallest set of tokens whose cumulative probability reaches P*. If `top_p=0.9`, sort tokens by probability descending, accumulate until you hit 90% of total probability mass, keep that set, discard the rest. When the model is confident (peaked distribution), you might keep 3 tokens to reach 90%. When the model is uncertain (flat distribution), you might need 200 tokens to reach 90%. This is more adaptive than top-K. But it has a subtle failure mode: on a genuinely flat distribution, the tail tokens that squeeze in just before the 90% threshold have very low individual probabilities — say, 0.1% each. They are statistically marginal but syntactically wrong. Top-P includes them anyway once the bulk of the probability mass is distributed across many tokens. Another common mistake: `top_p=1.0`. This includes the entire vocabulary. People set it thinking they are "disabling" top-P filtering, but all they have done is removed the filter entirely. With a moderate temperature, this means genuinely low-probability garbage tokens can appear. The correct way to disable top-P filtering is to not use it, or to use `top_p=1.0` only when paired with a near-zero temperature. ## Min-P: the 2024 upgrade that open-source runtimes adopted Min-P was proposed in 2024 and within a year became the default sampling method in llama.cpp, Ollama, and several other open-source runtimes, displacing top-P as the primary filter. The idea is to make the cutoff proportional to the top token's probability rather than to a fixed cumulative mass. Specifically: compute the top token's probability `p_max`. Set the minimum threshold at `min_p × p_max`. Discard any token whose probability is below this threshold. When the model is confident (`p_max = 0.80`), and `min_p=0.05`, the cutoff is `0.05 × 0.80 = 0.04`. Any token below 4% probability is dropped — a reasonably strict filter. When the model is genuinely uncertain (`p_max = 0.08`), the cutoff drops to `0.05 × 0.08 = 0.004`. Many tokens survive, reflecting the real ambiguity of the situation. ```mermaid flowchart TD PM["Top token probability p_max"] --> CALC["Threshold = min_p × p_max"] CALC --> FILTER["Keep tokens where p_i ≥ threshold"] FILTER --> RENORM["Renormalize remaining probabilities"] RENORM --> SAMPLE["Sample from filtered set"] style PM fill:#a855f7,color:#fff style CALC fill:#ffaa00,color:#0a0a0f style SAMPLE fill:#00e5ff,color:#0a0a0f ``` The result: min-P automatically adapts its strictness to the model's confidence at each decoding step. Top-P has a fixed target (90% of the probability mass); min-P has a relative target (be at least 5% as likely as the most likely token). In practice, min-P reduces nonsensical insertions on creative tasks — the kind where a story about a detective suddenly produces "banana" because top-P included low-probability tokens that happened to sit just inside the 95% cumulative window. Min-P discards those because they are nowhere near the probability of the next natural narrative token. A sensible starting point for creative work: `temperature=0.8, min_p=0.05`. For structured outputs: `temperature=0.1, min_p=0.05` (or just `temperature=0`, ignoring min-P entirely since greedy already takes the argmax). ## Parameter comparison | Method | What it cuts | Adapts to distribution? | Main failure mode | | --- | --- | --- | --- | | Greedy (T=0) | Everything except argmax | N/A | Repetitive, mode-collapsed output | | Top-K | All but top K tokens | No — fixed count | Too strict when model confident; too loose when uncertain | | Top-P | Tokens outside cumulative P mass | Partially — variable count | Includes low-p tail on flat distributions; p=1.0 is a footgun | | Min-P | Tokens below `min_p × p_max` | Yes — threshold scales with `p_max` | Setting min-p too high on uncertain distributions can over-prune | | Beam search | Non-beam paths | N/A (deterministic) | Computationally expensive; produces generic/averaged outputs | Beam search deserves a brief mention since it appears in older documentation: it maintains B candidate sequences in parallel at each step and keeps the top-B by cumulative log-probability. It is deterministic and tends to produce higher-likelihood but lower-diversity text. Modern practice has almost entirely moved away from it for generative tasks — the computational cost is `B×` the single-sequence cost and the outputs are worse for open-ended generation. It still shows up in constrained settings like machine translation with strict fluency requirements. ## The numbers: what these parameters cost you None of the sampling parameters add meaningful compute to the forward pass. The logit vector exists after the forward pass; filtering and sampling are O(V log V) sort operations on CPU or a simple parallel reduction on GPU — microseconds at most. The only sampling technique that has real latency implications is speculative decoding, and that is a *reduction*, not an increase. ``` Back-of-the-envelope: sampling overhead per token Forward pass for a 70B model at batch=1, A100 80GB: ~15-25ms per token (decode phase, memory-bandwidth bound) Sorting 128K logits for top-K filter: ~0.02ms (trivially parallelizable on GPU) Top-P or Min-P filter (linear scan after sort): ~0.001ms Total sampling overhead as fraction of forward pass: < 0.2% Conclusion: sampling parameters do not affect latency. The real latency lever is output token count, not sampling method. ``` The token count *is* affected by sampling choice in a subtle way: greedy decoding can fall into repetition loops that inflate output length. Filters like min-P and top-P break these loops by preventing the model from always returning to the same high-probability sequence. Setting a `repetition_penalty` (multiplicatively discounting already-generated tokens in the logit vector before softmax) is a complementary mechanism — not a sampling parameter per se, but usually mentioned alongside them. ## Reasoning models: where the API lies to you OpenAI's o1, o3, o4-mini, and Google's Gemini 2.0/2.5 Flash Thinking operate differently. These models generate extended internal reasoning traces before producing output. The sampling inside that reasoning chain is managed by the model and is not exposed through the API. Some providers accept a `temperature` parameter for compatibility but clamp it to a narrow internal range. Others return an error if you attempt to set it. A few silently ignore it. The model's output is not actually drawn from the distribution you think you are shaping. For these models, the useful parameters to tune are: - **`max_tokens` / `max_completion_tokens`**: controls how long the reasoning chain can run, which affects quality on hard problems - **`reasoning_effort`** (OpenAI o-series): controls thinking budget explicitly on some models - **Top-P, temperature**: treat as no-ops unless the provider documentation explicitly says otherwise This matters for cost calculations too. The internal reasoning tokens count toward your bill even though they are not returned in the response body (depending on provider). See [token economics](/ai-engineering/articles/token-economics-llm-api-cost-breakdown) for the billing mechanics. ## What breaks ### Greedy repetition loops The most common failure of greedy decoding on open-ended text: the model produces a sentence, and by the time it finishes the sentence, the most likely next token is the first word of the same sentence again. You get output like: "The solution is to check the configuration file. The solution is to check the configuration file. The solution is to check the configuration file." indefinitely. Greedy does not break these loops because the argmax is always the most probable token — and for a repetitive prefix, the most probable next token is more repetition. Adding a small temperature (0.2-0.4) or a repetition penalty breaks the cycle. ### Top-P undershooting on peaked distributions If `top_p=0.95` and the model assigns 97% probability to a single token, top-P keeps only that token (or maybe two). That sounds like greedy decoding, but it is not — you still sampled it. The issue is that on every *other* step where the model is uncertain, top-P may include problematic tail tokens. The parameter cannot be tuned to work well in both regimes simultaneously. ### Min-P passes (almost) everything on flat distributions With `min_p=0.1` and a very flat distribution where no token exceeds 2% probability, the threshold becomes `0.1 × 0.02 = 0.002`. Almost nothing gets pruned. This is correct behavior — min-P correctly diagnoses "the model is deeply uncertain; keep many candidates." But if your application needs *some* filter in this case (say, you need to avoid certain token classes), min-P alone is insufficient and you need an additional vocabulary constraint like logit biases or constrained decoding. The opposite failure exists too: set `min_p` too high and you over-prune — at `min_p=0.5` with `p_max=0.08`, only tokens above 4% survive, which on a genuinely flat distribution can leave one or two candidates and collapse the diversity you were sampling for. See [structured outputs and constrained decoding](/ai-engineering/articles/structured-outputs-constrained-decoding) for the production approach. ### Temperature=0 is not deterministic across hardware Multiple engineers have been surprised to discover that the same prompt with `temperature=0` produces different results on an A100 vs an H100, or even on the same GPU with a different batch size. The reason is floating-point non-associativity: changing the order of reduction operations in a matrix multiply changes the rounding errors, which can tip a close logit competition. The model you use in development (e.g., a hosted API) may use different hardware than your production inference server, producing different "greedy" outputs. If you need reproducible outputs, you need: fixed hardware type, fixed batch size of one, and a fixed random seed even with `temperature=0`. Providers rarely guarantee this across deployments. ### The creativity-factuality coupling High temperature increases creativity and increases hallucination simultaneously. The same mechanism that makes the model produce novel phrasings also makes it draw from lower-probability, less-grounded token sequences. There is no temperature setting that maximizes creativity while minimizing hallucination — they are directly coupled through the distribution geometry. The practical solution is to separate the tasks: use low temperature for factual extraction and grounding, use higher temperature for generation over retrieved facts. The [why models hallucinate](/ai-engineering/articles/why-llms-hallucinate-mechanistic) article covers the mechanistic reasons for this trade-off in detail. ## Speculative decoding: separate from but worth knowing alongside Speculative decoding is not a sampling parameter — it is an inference optimization that preserves your sampling distribution exactly. A small **draft model** (typically 1-7B parameters) generates K tokens autoregressively, then the **target model** (the one you actually want) verifies all K tokens in a single parallel forward pass. Each token is accepted or rejected based on whether the target model's distribution agrees. Accepted tokens carry through; the first rejected token is resampled from the corrected distribution. The math guarantees that the resulting token sequence is drawn from exactly the same distribution as if the target model had generated every token independently. Two to four times speedup at zero quality change — the settings you configure for temperature, top-p, and min-p on the target model still apply, and the output statistics are identical. See [speculative decoding](/ai-engineering/articles/speculative-decoding) for the mechanics. ## The decision in practice Pick your sampling configuration based on what kind of failure you cannot tolerate. **You cannot tolerate hallucination or format errors:** use `temperature=0` or `temperature=0.1`. Accept that outputs will be repetitive on some long generations; add `repetition_penalty=1.1` if that becomes a problem. Know where that knob exists, though: `repetition_penalty` is an open-source-runtime parameter (HF transformers, vLLM, llama.cpp) — OpenAI's closest equivalents are the additive `frequency_penalty`/`presence_penalty`, and Anthropic exposes neither, so on Claude the fix is a slightly higher temperature or a prompt-level instruction. For structured outputs, combine with constrained decoding — do not rely on temperature alone to enforce format. **You need diversity but care about coherence:** `temperature=0.7, min_p=0.05`. This is a reliable default for most chat and Q&A applications. The min-P filter handles distribution spikes better than top-P, and temperature=0.7 gives enough variance that responses do not feel canned. **You need creative range:** `temperature=0.9-1.1, min_p=0.03`. At this temperature range you will occasionally get odd outputs — that is the point. Run multiple samples and let your application logic or a downstream classifier select the best one. Best-of-N sampling at `temperature=0.8` frequently outperforms a single sample at `temperature=0` on creative tasks and on harder reasoning problems where the model benefits from exploring multiple paths. **You are calling a reasoning model:** set `temperature` to whatever the documentation says and ignore the section above. Set your output token budget thoughtfully — that is the only real lever you have on quality. ```python from anthropic import Anthropic client = Anthropic() # Structured extraction — near-deterministic. One sampling knob only: # the Anthropic API rejects requests that set both temperature and top_p — # which is fine, because stacking a filter on a temp-0.1 distribution # buys you nothing anyway. extraction_response = client.messages.create( model="claude-sonnet-4-5", max_tokens=512, temperature=0.1, # low, not zero — avoids hardware non-determinism issues messages=[{ "role": "user", "content": "Extract the invoice total from the following text as JSON: ..." }] ) # Creative generation — diversity wanted. Same rule: temperature OR top_p, # never both. min_p is not exposed here at all — it lives in open-source # runtimes like llama.cpp/Ollama. creative_response = client.messages.create( model="claude-sonnet-4-5", max_tokens=1024, temperature=0.9, messages=[{ "role": "user", "content": "Write the opening paragraph of a noir detective story set in 2045." }] ) ``` For open-source runtimes with min-p support (Ollama, llama.cpp, vLLM with recent builds): ```python import openai # vLLM exposes an OpenAI-compatible endpoint client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="ignored") response = client.chat.completions.create( model="meta-llama/Llama-3.1-8B-Instruct", messages=[{"role": "user", "content": "Write a short poem about debugging."}], temperature=0.8, extra_body={ "min_p": 0.05, # min-p passed as an extra field; not in OpenAI spec "repetition_penalty": 1.05, }, max_tokens=256, ) ``` The `extra_body` pattern for min-p works because most OpenAI-compatible servers pass through unknown parameters to the underlying sampler. Check your specific server's documentation — parameter names vary (`min_p`, `min_token_prob`, `min_probability`). If you are making decisions about which model to call for a given task — including whether to route to a reasoning model or a standard model — [model routing and cascades](/ai-engineering/articles/llm-model-routing-cascades-production) covers the production pattern. And if you are debugging why outputs feel repetitive or hallucinated, reading [why models hallucinate](/ai-engineering/articles/why-llms-hallucinate-mechanistic) and [tokenization and BPE](/ai-engineering/articles/tokenization-bpe-intuition) side by side is worth the hour — the roots of many output quality problems trace to either the sampling configuration or the way the prompt tokenizes, not to the model's capabilities. ## Frequently asked questions Q: What does temperature do in an LLM? A: Temperature is a scalar applied to the raw logit vector before softmax. Values below 1.0 make the distribution sharper — the highest-probability token is chosen more often. Values above 1.0 flatten the distribution, giving low-probability tokens more chance to be selected. Temperature 0 collapses to greedy decoding (always pick the argmax), but even at T=0 hardware non-determinism can produce slightly different outputs across runs. Q: What is the difference between top-p and min-p sampling? A: Top-p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability sums to p, then samples from that set. Min-p sets a per-step threshold as a fraction of the top token's probability — if the top token has 40% probability and min-p=0.05, any token below 2% is cut. Min-p adapts dynamically: when the model is confident (peaked distribution) the cutoff is strict; when uncertain (flat distribution) more candidates survive. This produces fewer nonsensical insertions than top-p on creative tasks. Q: When should I use greedy decoding vs sampling? A: Use greedy decoding (temperature=0 or temperature very low) for structured outputs like JSON extraction, code generation, and classification where you need the single most-likely token at every step. Use sampling (temperature 0.7-1.0 with top-p or min-p) for creative writing, brainstorming, and conversational text where diversity improves output quality. For reasoning models the picture varies by provider: OpenAI o-series models reject or ignore temperature, while others (e.g. Gemini thinking models) accept it — but the internal reasoning sampling is not exposed either way, so check the provider docs before tuning it. Q: What is the right temperature for code generation? A: For production code generation, temperature in the range 0.0 to 0.3 works well. Lower temperatures favor syntactically correct, predictable output. For exploration (generating multiple candidate implementations to pick from), a temperature around 0.7 with top-p=0.95 or min-p=0.05 increases variety without producing incoherent garbage. Many teams run temperature=0 in CI pipelines and temperature=0.7 in interactive coding assistants. Q: Does top-p=1.0 disable sampling? A: No — top-p=1.0 includes the entire vocabulary in the candidate set, meaning every token with any nonzero probability can be sampled. This does not make outputs deterministic; it makes them noisier. To get deterministic (greedy) outputs, set temperature=0 or top-k=1. Top-p=1.0 combined with a moderate temperature is a common source of unexpectedly incoherent outputs. Q: What is speculative decoding and how does it relate to temperature? A: Speculative decoding is a latency technique: a small draft model proposes several tokens in sequence, then the target model verifies or rejects them in a single forward pass. It is quality-neutral — the same temperature and sampling parameters apply to the verification step, so outputs are statistically identical to normal decoding at the same settings. Production deployments report 2-4x latency reduction without changing output distribution.