The Model Landscape: Choosing Without Getting Played

Two teams at the same company pick models for a new AI product. Both spend an afternoon on the leaderboard, both point at the same chart showing their chosen model hitting 89.3 on MMLU. Six months later, one team has a product in production. The other is rewriting everything because the model they chose costs four times what they projected, gets deprecated mid-launch, and performs terribly on the specific tasks their users actually send. The leaderboard looked identical. The outcomes weren't.

This module is about not being that second team.

The model selection decision sounds like it should be simple — go to a leaderboard, find the highest number, use that. But that framing hides three real problems: benchmarks routinely misrepresent what models actually do on your task, the open versus closed calculus involves at least five dimensions that aren't on any leaderboard, and models you choose today may not exist in the same form twelve months from now. You need a framework, not a finger-point.

The shape of the current model map

As of mid-2026, the model market has settled into three distinct tiers. Understanding what each tier is optimized for — and what it costs — is the foundation of every model selection decision.

flowchart TD
    A[Your use case] --> B{Complexity?}
    B -->|Standard tasks| C[Standard frontier tier]
    B -->|Hard reasoning needed| D[Reasoning model tier]
    C --> E{Data constraints?}
    E -->|Data stays in VPC| F[Open-weight self-hosted]
    E -->|API acceptable| G[Closed frontier API]
    D --> K{Data constraints?}
    K -->|Data stays in VPC| H["Self-hosted DeepSeek-R1 class"]
    K -->|API acceptable| L["o3 / Extended thinking / Deep Think"]
    F --> I[Llama 4 / DeepSeek V4 / Mistral 3]
    G --> J[GPT-5.x / Claude Sonnet 4.x / Gemini 3.1 Pro]
    style A fill:#00e5ff,stroke:#00e5ff,color:#0a0a0f
    style H fill:#a855f7,stroke:#a855f7,color:#fff
    style L fill:#a855f7,stroke:#a855f7,color:#fff
    style I fill:#0e7490,color:#fff
    style J fill:#0e7490,color:#fff

At the top of the API market sit the closed frontier models — GPT-5.x, Claude Opus/Sonnet 4.x, Gemini 3.1 Pro — the best-performing models you can reach without owning a single GPU. They handle multimodal inputs (text, images, some handle audio and video) and come with the operational convenience of managed services. Pricing in mid-2026 runs roughly $3–15 per million output tokens for frontier-tier models — illustrative; check vendor pricing pages since rates have fallen 30-60% from 2024 to mid-2026.

If your data can't travel, or your volume is enormous, you look instead at open-weight models (Llama 4 Scout/Maverick, DeepSeek V4, Mistral Large 3, Qwen 2.5/3) — weights you download and deploy yourself. The performance gap versus closed models has closed to near-zero on knowledge benchmarks and sits at 5-10 points on complex agentic tasks as of mid-2026. What you get is data sovereignty, the ability to fine-tune on proprietary data with full control, and a cost structure that improves with volume. What you take on is real operational burden.

The third tier is different in kind. Reasoning models (OpenAI o3, DeepSeek-R1, Claude Opus 4.x with extended thinking, Gemini 3 Deep Think) use test-time compute to think before answering — an internal chain-of-thought, sometimes thousands of tokens, generated before the response you see. They are genuinely better on multi-step math, competitive programming, complex research tasks, and anything where a standard model reliably makes logical errors. They are also 3-10x slower and costlier per task. DeepSeek-R1, released in January 2025, proved this capability could emerge from pure reinforcement learning without supervised fine-tuning, at roughly 96% lower API cost than o1 at the time — the most significant demonstration that reasoning is a training approach, not a proprietary secret.

Within the first two tiers, there's also a size spectrum inside open-weight models: 3B–8B "edge" models run on a single consumer GPU, handle classification, extraction, and summarization at low cost; 30B–70B "mid-tier" models handle most standard generation tasks; 400B+ MoE frontier models (DeepSeek V4, Llama 4 Maverick) need distributed serving at serious infrastructure cost but match closed API quality. The wrong size for a task wastes either quality or money.

Why leaderboards mislead you

Here is a number: 89.3. That's a hypothetical MMLU score. Here's what it doesn't tell you: whether the training data overlapped with the MMLU test set, what the model does on your specific document type, how it behaves at the edges of its context window, what version of the model that score was measured on, or whether the evaluation was conducted with or without chain-of-thought.

MMLU has been saturated. The top twenty models cluster within a 3-4 point band, which is within the noise of prompt formatting choices and temperature settings. Treating any delta under five points as a real capability difference is a methodological error.

There are two main contamination vectors. The first is direct: models trained on web-scale text often contain the benchmark questions themselves, because those datasets live on the web. The second is subtler: models trained to score well on public benchmarks via RLHF or preference optimization will optimize for the evaluation patterns specific to that benchmark, not for general capability.

LiveBench is an attempt at a partial fix — it rotates questions from recent problems (competition math, recent news, new code challenges) so training contamination is lower. Chatbot Arena uses blind human preference comparisons, which at least measures what people prefer rather than what scores well on academic tests. Neither substitutes for your own eval.

The benchmark literacy rule: use public benchmarks for rough tier-level comparisons (is this model in the frontier tier or not?), and use your own held-out eval set for anything that actually affects which model you deploy.

Build your eval set first

Before you even open an API playground, spend two hours building a 50-example labeled set from your actual use case. This sounds tedious. Do it anyway. It is the highest-return thing you can do before spending engineering months on a choice that locks in production behavior.

What goes in a 50-example eval set:

• 20 representative examples of the most common request type you'll receive.
• 10 examples from the "hard end" — edge cases, ambiguous inputs, unusual formatting.
• 10 examples where you know the right answer (golden outputs you can score deterministically).
• 10 examples that probe the specific failure modes you're most worried about (hallucination on your domain, refusals on sensitive but legitimate tasks, degradation at long context).

Run this set through each model you're considering, score the outputs on a simple 1-3 scale, and compute the average. You'll get a signal that actually reflects your task within a few hours. This is your test drive. The leaderboard was the brochure.

import anthropic
import openai
from typing import Callable

# Illustrative eval harness — adapt to your task
def run_eval(
    model_fn: Callable[[str], str],
    examples: list[dict],  # [{"input": "...", "ideal": "..."}]
    score_fn: Callable[[str, str], int],  # returns 1-3
) -> float:
    scores = []
    for ex in examples:
        output = model_fn(ex["input"])
        scores.append(score_fn(output, ex["ideal"]))
    return sum(scores) / len(scores)

# Wire up two models
def gpt_fn(prompt: str) -> str:
    client = openai.OpenAI()
    resp = client.chat.completions.create(
        model="gpt-4o-2024-11-20",  # pin model version
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.choices[0].message.content

def claude_fn(prompt: str) -> str:
    client = anthropic.Anthropic()
    resp = client.messages.create(
        model="claude-sonnet-4-5-20250929",  # pin model version
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return resp.content[0].text

Pin the model version string. Always. Model providers update the behavior of un-versioned aliases (like gpt-4o without a date suffix) without warning. The model you evaluated in your eval harness should be the exact model you deploy.

The open versus closed decision

This is not a philosophical stance. It's an engineering tradeoff with five concrete axes.

flowchart LR
    subgraph "Closed API"
        C1[No infra ops]
        C2[No GPU cost]
        C3[Data leaves VPC]
        C4[Vendor lock-in]
        C5[No weight access]
    end
    subgraph "Open-weight self-hosted"
        O1[Full infra ops burden]
        O2[GPU cost at volume]
        O3[Data stays in VPC]
        O4[No lock-in]
        O5[Fine-tune freely]
    end

Cost floor and ceiling. Closed APIs bill per token. At low volume, this is cheap and predictable. At high volume — above roughly $5,000/month — the math starts to favor self-hosting an open-weight model, but that math only works if you account for GPU instance cost, DevOps time, on-call burden, and the latency overhead of managing your own inference cluster. Deploying Llama 4 Maverick at production scale needs at minimum 4-8 H100s. That's not a weekend project.

Data residency. This is often the decision-maker in regulated industries. Healthcare (HIPAA), finance (various), defense (obvious), and European enterprises subject to strict GDPR interpretations often cannot send customer data to a third-party API. If your data cannot leave your VPC, open-weight self-hosted is not an option to consider — it's the only path.

Fine-tuning control. Both OpenAI and Anthropic offer fine-tuning APIs, but they give you limited visibility into the training process and no access to intermediate checkpoints. Open-weight models let you run SFT, DPO, or RL-from-scratch on proprietary data with full control. When task-specific fine-tuning yields more than 10 percentage points of quality improvement on your eval set, that control starts mattering. Below that threshold, prompt engineering and RAG almost always win on simplicity. One caveat that hits exactly the regulated-industry teams who need open weights most: "open-weight" is not one license. Mistral ships Apache-2.0; Meta's Llama community license carries a 700M-monthly-active-user clause and an acceptable-use policy; some releases are research-only. Have legal read the actual license before you commit GPUs — not every open weight is open for your use case.

Deprecation and versioning risk. Closed providers retire model versions. OpenAI and Anthropic typically give 6-12 months of notice, but "typically" is doing a lot of work in that sentence. If you build a product on a model that gets deprecated, you have to re-eval and re-prompt against the replacement — and the replacement behaves differently. Pin your model version string, subscribe to deprecation announcements, and have a migration test suite ready. Open-weight models don't get deprecated in the same way — the weights you download today still run tomorrow.

Multimodality. All the major closed frontier models handle images natively. GPT-5.x and Gemini 3.1 Pro handle audio and video inputs. Open-weight options here are improving — Llama 4 is among the first mainstream open-weight series trained natively multimodal from pretraining rather than bolted-on (Meta's own Chameleon and Adept's Fuyu-8B got there earlier with less fanfare) — but coverage varies significantly by model version and modality type. Always check the actual API documentation for which modalities are live, not what the model announcement said. Marketing often outpaces the deployed API by months.

{ "type": "model-router", "title": "Routing decisions: when cheap and when frontier" }

The router visualization above illustrates something important: a lot of production systems shouldn't be using one model for everything. Simple classification, intent detection, and extraction tasks often run adequately on models an order of magnitude cheaper than frontier. Only the hard cases need the expensive one. This is model routing — covered deeply in Model Routing and Cascades — and it's often worth more than any single model choice.

Size tiers and what they're actually for

Inside the open-weight tier especially, model size is a first-order variable.

Size tier	Parameters	Typical use	GPU requirement
Edge	1B–8B	Classification, extraction, summarization, assistants on device	1× consumer GPU or CPU
Mid	14B–32B	RAG generation, chat, translation, code assist	1-2× A10G/A100
Large	70B–90B	Complex reasoning, long-form generation, agentic tasks	4× A100
MoE frontier	400B+ total (sparse MoE, ~17–40B active)	GPT-4-class tasks, competitive coding, research	8× H100+

A fine-tuned 8B model often outperforms a general-purpose 70B model on a narrow domain, at one-tenth the inference cost. If your task is well-defined and you have domain data, that's worth investigating before defaulting to the biggest model in the catalog.

The "just use the biggest model" instinct is expensive and often wrong. The "use the smallest model that meets quality bar on your eval set" instinct is the one that doesn't get you paged for a $200K bill.

Reasoning models: when the slowdown is worth it

Standard models generate one token at a time with no explicit deliberation. Reasoning models spend compute before the visible output, thinking through the problem in a scratchpad. The result is better performance on tasks that require multi-step logic — but the latency and cost increase is not marginal.

Use a reasoning model when: your task involves multi-step math or formal logic where standard models make mistakes, the task is high-stakes enough that a 3× cost increase is acceptable, and you have evaluated a standard model on your 50-example set and found it demonstrably fails the hard cases.

Do not use a reasoning model for: text classification, extraction, summarization, FAQ answering, or any task where a standard model gets 90%+ on your eval set. The overhead is pure waste.

{ "type": "token-cost", "title": "What a month of output tokens costs at each model tier" }

One more thing about reasoning models: the "extended thinking" or chain-of-thought traces they produce can be extremely long — thousands of tokens — and that length counts against your context budget and your bill. A task that yields a 300-token answer from a standard model can spend 3,000-10,000 thinking tokens getting there on a reasoning tier, all billed at output rates — that's where the 3-10x cost multiple comes from, not the visible answer. Some providers charge differently for thinking tokens versus output tokens. Read the pricing page for the specific model, not the general category.

Multimodality: check before you build

The modality situation in mid-2026 is messy and fast-moving. The marketing materials describe what a model was trained on; the API documentation describes what you can actually call today. These are not always the same thing.

GPT-5.x, Gemini 3.1 Pro, and Llama 4 handle images natively and well. Native audio input and output is live for GPT-5.x and Gemini; Claude 4.x models as of mid-2026 remain text and image only for API inputs, though that may change. Video input is live for Gemini but more limited elsewhere.

Cross-modal hallucination is a known and unsolved problem. A model asked to describe an image will sometimes describe content that isn't there, with the same confident fluency as its text outputs. The hallucination mitigation techniques in Hallucination Mitigation and Output Validation apply, but the baseline rates are higher than text-only tasks.

For document AI specifically — PDFs, scanned forms, tables — Document AI and OCR in 2026 covers when vision-language models beat classic OCR engines and when they don't. The answer is less obvious than it sounds.

What actually breaks in production

The model selection mistakes that cause the most pain after launch tend to cluster around four failure patterns.

The most common is the benchmark-to-reality gap. A team picks a model that scores 91.2 on some aggregate benchmark. In production, on the specific task of extracting structured fields from messy customer emails, it's wrong 15% of the time — which their eval set would have caught in two hours. They discover this at 2am three weeks post-launch.

Then a model version gets sunset — deprecation mid-product. The replacement version behaves differently on edge cases: outputs that were reliably JSON-formatted now occasionally return prose, or refusal patterns shifted. If you didn't pin the model version and build a migration test suite, this surprise hits you in production.

Cost projections fail more quietly. The "we'll just use GPT-frontier-tier" assumption runs into the actual traffic volume, and the back-of-envelope math is worth doing before launch rather than after:

API bill (frontier tier, illustrative $10/M output tokens):
  500 req/day    × 2,000 output tok =   1M tok/day →    ~$10/day →    ~$300/mo
  50,000 req/day × 2,000 output tok = 100M tok/day → ~$1,000/day → ~$30,000/mo

Self-hosting the same volume (Llama 4 Maverick class):
  8× H100 reserved ≈ $15-20K/mo, plus ~0.5 engineer of serving ops
  → GPUs + ops can't beat a $300/mo API bill; they can beat a $30K one.
    The crossover lands around a few $K/mo of API spend — hence
    the ~$5K/mo rule of thumb.

The first line is a rounding error; the second is a headcount decision. Building a cost model before launch is not optional for any product with meaningful volume. The Token Economics article walks through the math.

sequenceDiagram
    participant Dev as Developer
    participant Bench as Leaderboard
    participant Prod as Production
    Dev->>Bench: Pick highest number
    Bench-->>Dev: Model X scores 91.2
    Dev->>Prod: Deploy Model X
    Prod-->>Dev: 15% error rate on real task
    Note over Dev,Prod: Real task ≠ benchmark task
    Dev->>Dev: Build private eval set (should have done this first)
    Dev->>Prod: Re-evaluate 3 models
    Prod-->>Dev: Model Y scores best on actual task

The fourth pattern lives on the other side of the fence: the open-weight operational underestimate. A team decides to self-host to save cost. They didn't account for the engineering time to set up vLLM or SGLang, manage GPU health, handle model loading failures, implement continuous batching, and monitor KV cache utilization. Six months in, they've spent more on engineering time than the API would have cost. The Serving Engine Comparison gives the honest picture of what self-hosting actually involves.

A practical selection checklist

Before committing to a model:

• Run your 50-example eval set on at least two models. Score with a simple rubric. Do not skip this step.
• Compute your projected monthly cost at expected traffic. Multiply your estimate by 3× for safety margin on output tokens (users always send more than you expect).
• Check the model's documented context window behavior. A 128K context window does not mean quality is uniform across 128K — the lost-in-the-middle effect degrades recall for information in the middle of long contexts. Context Windows, KV Cache, and Reality covers the mechanics.
• Check actual API modality support against your requirements. Not the announcement — the API docs.
• Identify the model version string you'll pin. Confirm deprecation timelines if available.
• If you're considering open-weight: honestly estimate GPU cost, DevOps time, and on-call burden. If the number is uncomfortable, use the API.

{ "type": "eval-pipeline", "title": "Where private evals fit in the safety net" }

For anything beyond the simplest classification task, also read Building the Safety Net Before You Need It. Model selection and eval strategy are tightly coupled — choosing the right model for your eval set only works if your eval set actually measures what matters.

Where to go next

The Model Landscape in 2025-2026 — the deep reference on specific models, their benchmark positions, pricing, and capability gaps. Use this when you need specifics for a model choice you're making today.

Reading LLM Benchmarks Without Getting Fooled — full treatment of contamination, saturation, and which benchmarks are actually informative for which task categories.

Open vs. Closed Models: A Decision Framework — all five decision axes (cost, data residency, fine-tuning, ops burden, performance) with the actual math worked out.

Model Routing and Cascades — how to build a system that uses cheap models for easy tasks and expensive models only where needed. Often worth 3-5× cost reduction over single-model systems.

Eval First: Building the Safety Net — the eval methodology to build before you make any model decision. You should probably read this first.

Cost Control and Reliability Engineering — what happens after you've chosen a model and need to stop the bill from surprising you.

Reasoning Models: When and How to Use Them — the full decision framework for the reasoning tier, with worked examples of tasks where the cost and latency overhead is and isn't justified.