Open vs. closed models: a decision framework, not a religion

Three days after you demo your new support assistant to the VP of Product, someone from legal shows up with a question: does customer message data leave your infrastructure when it hits the model API? You check, and yes — every token you send to the closed frontier API goes to a third-party data center under a standard SaaS agreement that your legal team has not reviewed. The feature freeze lands a week later.

That scenario plays out constantly. Teams pick a model based on capability and forget that the decision has a data governance dimension, a cost trajectory dimension, and an operational burden dimension. By the time those constraints surface, you've shipped integrations that are expensive to change. Making this call deliberately, before you build, is the first real engineering decision of any AI project.

What actually changed in 2025–2026

The open/closed debate looked different two years ago. Closed frontier models (GPT-4 class) held a commanding 15–20 point lead on most benchmarks. Open-weight models topped out around the Llama 2 70B range — capable but clearly not in the same tier. Self-hosting was a clear cost-efficiency play for teams that could afford the infrastructure.

Three things broke that picture.

DeepSeek-R1 (January 2025) proved that state-of-the-art reasoning capability can emerge from pure reinforcement learning — no proprietary supervised fine-tuning data required. It matched OpenAI o1 on AIME 2024 (the hardest public math benchmark at the time), released weights under an MIT license, and launched an API at roughly 96% lower cost than o1. The model shipped; the field recalibrated.

Meta Llama 4 (April 2025) introduced genuinely multimodal Mixture-of-Experts open-weight models trained jointly on text, images, and video. Scout — the smaller variant — supports a 10-million-token context window. Maverick is competitive with GPT-4o on most benchmarks. Neither is a fine-tuned text model with a vision bolt-on; both were trained multimodal from pretraining. This was the first time open-weight and closed frontier models were genuinely comparable on modalities.

Pricing compression accelerated throughout 2025–2026. As of mid-2026, GPT-4-class output tokens run $3–15 per million depending on the model and tier, down 30–60% from 2024. The cost argument for self-hosting is weaker at low-to-moderate volumes than it was two years ago — not because self-hosting got more expensive, but because the alternative got cheaper.

The result: on standard knowledge and language benchmarks (MMLU, HellaSwag, ARC), the performance gap between open-weight and closed frontier is functionally zero as of early 2026. The gap persists on complex agentic tasks and production-scale coding (SWE-bench), where closed models still hold 5–10 points. Whether that delta matters for your workload is the crux of the decision.

The four dimensions of the decision

1. Data governance

This one is binary. Either your data can leave your infrastructure or it cannot.

Financial services firms operating under strict data residency requirements, healthcare organizations with patient data covered by HIPAA (regardless of BAAs), defense contractors working with controlled unclassified information, and companies in jurisdictions with strict data localization rules (EU under some interpretations of GDPR for certain categories) often cannot send data to a third-party API — not because they haven't tried, but because their legal counsel has reviewed the contracts and said no.

For those organizations, the choice is made for them: open-weight model on private infrastructure, or managed open-weight inference from a provider that runs entirely within their cloud account (Azure's serverless Llama deployments, AWS Bedrock's Llama access, or self-operated clusters on GCP/AWS). The closed vs. open performance debate is academic when the data constraint is absolute.

For everyone else: most closed providers offer enterprise agreements with data processing addendums, SOC 2 Type II certification, HIPAA BAAs (OpenAI, Anthropic, Google each offer these for enterprise tiers as of mid-2026), and optional zero-data-retention modes. Check what your legal team actually needs before assuming you cannot use the API — many teams discover the closed API is fine with the right contract in place.

2. Cost — the math that teams consistently get wrong

The most common mistake is comparing per-token API pricing to bare-metal compute cost and concluding that self-hosting is obviously cheaper. It usually is not, at moderate volumes, once you account for everything.

Full cost of self-hosting Llama 4 Maverick (4× H100 SXM, fp8):

Hardware/cloud:
  4× H100 SXM (80GB NVLink) on spot: ~$3.00/hr each = $12.00/hr
  On-demand (reserved): ~$5.50/hr each = $22.00/hr
  Monthly (spot, 730hr): $8,760
  Monthly (reserved): $16,060

Throughput (realistic at 70% utilization):
  ~1.5–2B tokens/month output at standard batch sizes
  Effective per-M-token cost (spot): $4.38–5.84/M
  Effective per-M-token cost (reserved): $8.03–10.71/M

Engineering:
  1 SRE/MLOps engineer managing the cluster: ~$15K/month fully-loaded
  Spread over multiple workloads on the same cluster, this is amortized.
  On a single workload, it dominates.

Hidden costs:
  Security patching of model weights and inference stack
  Monitoring, alerting, oncall rotation
  Network egress if serving to external users
  Version management when better checkpoints release

Break-even against a $10/M API:
  Token volume where API cost equals (compute + 1 engineer):
  ($8,760 + $15,000) / $10/M × 1M = 2.38B tokens/month minimum
  → You need >2B output tokens/month before self-hosting wins,
    assuming you spread engineering cost across the cluster.

The number shifts if you already have a GPU cluster for other workloads, if you can run multiple models on the same hardware, or if your team has the MLOps capacity built in. But teams doing back-of-envelope cost comparisons that exclude engineering overhead routinely underestimate self-hosting cost by 3–5×.

The practical threshold: if your projected monthly API spend is below $5K, the closed API is almost certainly cheaper in total cost of ownership. Above $10K, the conversation is worth having. Above $30K, self-hosting or managed open-weight inference is likely the right call — assuming you have or can hire the operational expertise.

{ "type": "token-cost", "title": "API vs self-hosting cost model" }

3. Performance — where the gap actually lives

The flat claim that "closed models are smarter" is too coarse to be useful. The gap is task-dependent, and by mid-2026 it is smaller than most teams assume.

flowchart LR
    subgraph "Gap near-zero (early 2026)"
        K[Knowledge Q&A\nMMKU, TriviaQA]
        S[Summarization]
        CL[Classification]
        EX[Extraction\nNER, schema fill]
        TR[Translation]
    end
    subgraph "Gap persists (5-10 points)"
        AG[Complex multi-step\nagent tasks]
        CD[Production coding\nSWE-bench verified]
        MA[Hard math\nAIME, MATH-500]
        SC[Long-context\nreasoning >100K tokens]
    end

    style K fill:#15803d,color:#fff
    style S fill:#15803d,color:#fff
    style CL fill:#15803d,color:#fff
    style EX fill:#15803d,color:#fff
    style TR fill:#15803d,color:#fff
    style AG fill:#00e5ff,color:#0a0a0f
    style CD fill:#00e5ff,color:#0a0a0f
    style MA fill:#00e5ff,color:#0a0a0f
    style SC fill:#00e5ff,color:#0a0a0f

For customer support classification, document summarization, information extraction, and single-turn Q&A on well-defined domains, a well-prompted Llama 4 Scout or Mistral Large 3 is competitive with GPT-4o and Claude Sonnet 4 on most held-out evaluations. The marginal quality gain from the closed frontier model often doesn't justify the cost, especially at volume.

For multi-step agentic tasks — the model is calling tools, writing code, executing it, debugging, and iterating over several steps — the closed frontier models still have a measurable lead. This is not primarily a benchmark artifact; it shows up in production error rates on real agentic pipelines. The leading closed models handle ambiguous intermediate states better, recover from tool errors more gracefully, and produce more reliable code on the first attempt.

The lesson: run your own evaluation on your actual task before choosing. Public benchmark scores are directional at best. Reading leaderboard scores without getting fooled is its own skill — contamination, benchmark-specific optimization, and methodology differences make cross-model comparisons unreliable unless you've run on your own held-out data. Benchmark contamination in training data is a well-documented phenomenon; a model that scores 90% on MMLU may have seen MMLU examples during pretraining.

4. Fine-tuning potential

Open-weight models can be fine-tuned on proprietary data using SFT, DPO, or GRPO. Closed models offer prompt-level adaptation and, in the case of OpenAI and Anthropic, limited fine-tuning APIs — but you don't own the resulting checkpoint, can't run it on your own hardware, and have limited control over training data handling.

Fine-tuning matters most for narrow, stable, high-volume tasks where a domain-specific model meaningfully outperforms a general-purpose model. Common examples: medical coding classification, legal clause extraction, company-specific document Q&A, e-commerce product attribute extraction.

A fine-tuned 8B open-weight model routinely outperforms a zero-shot 70B model on tasks where the training data matches the target distribution. The 10× parameter reduction translates directly to inference cost and latency. But the fine-tuning process has a real cost: data collection and quality control (often the dominant investment), training compute, evaluation, and ongoing maintenance as the base model evolves. See the fine-tuning decision framework before committing to this path — the question of whether fine-tuning, RAG, or better prompting is the right tool deserves its own analysis.

The managed open-weight tier (the overlooked middle)

Most teams think the choice is "closed API" vs. "self-hosted open-weight on our own cluster." There is a third option that solves the data governance problem without requiring you to own the infrastructure: managed open-weight inference.

Providers like Together AI, Fireworks AI, Baseten, and cloud-native offerings (Azure serverless Llama, AWS Bedrock Llama, Google Cloud partner models) let you call Llama 4, Mistral, or DeepSeek via an API endpoint that runs in a specific region or cloud account. Some configurations are single-tenant or VPC-hosted, satisfying tighter data residency requirements. Pricing is typically 30–60% below equivalent closed-model APIs for similar capability tiers, with no cluster management.

This tier is underused. Teams default to either the branded closed APIs (OpenAI, Anthropic, Google) or full self-hosting; they overlook managed open-weight as a path to lower cost without the operational burden. It's worth pricing these options before committing to a self-hosted cluster.

What breaks

Assuming benchmark parity means task parity

The performance gap closing on MMLU does not mean Llama 4 Maverick and GPT-5 are interchangeable. Benchmarks measure what they measure. Agentic failure modes — getting stuck in loops, misunderstanding ambiguous tool outputs, producing syntactically valid but semantically broken code — show up in production before they show up in standardized evals. Don't migrate a critical agentic workflow from a closed model to an open-weight model without running the two head-to-head on your actual task at scale.

Underestimating the operational tax of self-hosting

Self-hosted LLMs are not "set it and forget it." Base model checkpoints get updated; the open-source inference stack (vLLM, SGLang) ships breaking changes; CUDA driver updates interact with quantization libraries in non-obvious ways. Security patching for model weights is not a solved problem — there is no automatic update mechanism for a 400B-parameter checkpoint the way there is for a Docker image. Someone on your team owns this. Make sure that someone exists before you commit to a self-hosted path.

Mistaking "open-weight" for "open-source"

Most major "open-weight" models are not fully open-source. Meta's Llama license restricts commercial use for services with over 700 million monthly active users. DeepSeek's MIT license is more permissive, but the model was trained in China, which introduces separate considerations for some organizations around supply chain trust. Mistral is one of the cleaner open licenses in the current field. Read the actual license before making architectural decisions that assume you can do anything you like with the weights.

Vendor lock-in in both directions

The obvious lock-in risk is with closed APIs: proprietary prompting idioms, tool-call formats, and model-specific behavior that doesn't transfer. But open-weight self-hosting creates its own lock-in: operational investment in a specific serving stack, fine-tuned checkpoints tied to a specific base model version, and organizational muscle memory around a particular deployment pattern. Neither is free. The answer is to use provider-agnostic interfaces (LiteLLM, the OpenAI-compatible API surface that most serving engines expose) from day one — see model routing and cascades for patterns that let you swap models without re-architecting.

Treating the choice as permanent

Model capabilities and pricing move fast. A team that committed to a closed API in 2023 for capability reasons may find that the open-weight tier now matches their quality requirements at 40% of the cost. A team that self-hosted aggressively in 2024 to save money may find that 2026 API pricing makes the managed option competitive again. Instrument cost and quality from the start — per-model, per-task, per-dollar — so you can make the migration case when the numbers change. LLM observability and monitoring tooling should record not just output quality but model provenance and cost per inference.

A comparison of model serving paths

Path	Time to first token	Monthly cost (500M output tokens)	Data governance	Fine-tuning	Operational burden
Closed API (GPT-5, Claude Opus 4.x)	Hours	~$2.5K–7.5K	BAA/DPA available	Limited API	None
Managed open-weight (Together, Fireworks)	Hours	~$1K–3K	Region/VPC options	No	Minimal
Cloud-native open-weight (Azure Llama, Bedrock)	Hours	~$1.5K–4K	Cloud account boundary	No	Minimal
Self-hosted open-weight (vLLM on H100s)	Weeks	~$3K–9K + engineer	Your VPC	Yes	High
Fine-tuned self-hosted	Months	~$3K–9K + fine-tune cost	Your VPC	Yes (done)	Very high

Costs are illustrative for mid-2026 pricing and will drift. The relative ordering is more stable than the absolute numbers.

The decision in practice

Here is the framework as a concrete sequence of questions, not a flowchart you look at once and ignore.

First: can your data leave your infrastructure? If no, the choice is open-weight, and the remaining questions are about which serving path and whether fine-tuning is warranted. If yes, continue.

Second: what is your projected monthly token volume? Estimate realistically: number of requests per day × average tokens per request (input + output). If the resulting API cost at closed-model rates is below $3K/month, use the closed API. Do not self-host at this volume. The engineering overhead is not worth it.

Third: what does your task-specific evaluation show? Run both the cheapest closed model (a Sonnet-tier or Flash-tier, not the frontier flagship) and a capable open-weight model on your actual held-out data. Many tasks don't require frontier performance. If the cheaper tier passes your quality bar, you've just cut cost without touching infrastructure.

Fourth: if cost is a forcing function at higher volumes, evaluate the managed open-weight path before self-hosting. Together AI, Fireworks, and the cloud-native offerings have meaningfully simplified this. You get open-weight pricing without the cluster.

Fifth: consider fine-tuning only when you have a narrow, stable task, at least a few thousand high-quality labeled examples, a held-out eval that shows a meaningful quality gap, and an organization that can maintain the checkpoint as base models evolve. Fine-tuning is a commitment, not a configuration option.

The one thing to avoid is making this call based on principle rather than numbers. "We believe in open-source" is not an engineering argument. "Our legal team says data cannot leave our AWS us-east-1 account" is. "Self-hosting is more serious" is not. "Our cost projection is $40K/month at current API rates and we have an ML platform team" is. Stay in the numbers.

{ "type": "model-router", "title": "Cost vs quality routing across model tiers" }

The picture will keep shifting. Closed model pricing will probably keep falling. Open-weight quality will keep improving. The right answer today may not be the right answer in 18 months. What doesn't change is the decision structure: governance first, then cost math, then quality evidence, then operational capacity. Run through those in order every time the picture changes, and you'll make the call correctly regardless of what the models look like when you get there.