Build vs Buy and Small vs Large: The Framework for Model and Infrastructure Decisions

The bill came in at $23,000 for October. The team had launched a document-processing feature in September, routed every call to Claude Sonnet 4, and never looked at the per-feature breakdown. An afternoon of cost attribution in the gateway revealed that 71% of spend was on a bulk extraction pipeline running overnight that had no SLA requirement. Moving that pipeline to Anthropic's batch API (roughly 50% off) and switching the simple extraction step to a fine-tuned 7B model cut the pipeline's ~$16,300 to about $3,000. The November bill came in at $9,700. No model quality changed. No user noticed.

That story repeats itself constantly. The build-vs-buy question and the small-vs-large question are not architecture decisions made once — they are continuously wrong until someone actually measures what the bill is buying.

The two-axis decision space

The build-vs-buy and small-vs-large decisions are orthogonal but interact in practice. You can:

• Use managed APIs with frontier models (easy, expensive at scale)
• Use managed APIs with small models via providers like Groq or Together AI (easy, cheap, latency-constrained by network)
• Self-host large models (expensive infrastructure, maximum quality control)
• Self-host small models (the cheapest path, but operationally non-trivial)

There is also a middle tier the binary framing hides, and it is where many real decisions land: managed dedicated capacity. Provisioned throughput on Bedrock, Azure PTUs, and dedicated open-weight endpoints from Fireworks, Baseten, Modal, or Together give you a flat per-GPU-hour or per-unit rate on hardware someone else operates. You get most of the open-weight serving economics — often within 20–40% of raw GPU cost — without the 0.05–0.15 FTE ops tax discussed below, because the provider handles the serving stack, model updates, and 3 AM pages. Above roughly $2,000–3,000/month of steady, predictable traffic, this tier frequently beats pay-per-token; it loses to true self-hosting only at high volume with strong in-house infra skills, and to per-token APIs when traffic is spiky enough that dedicated capacity sits idle.

The right answer depends on where you are on each axis — and that changes as your product scales. What is correct at $500/month of API spend is almost certainly wrong at $50,000/month, and vice versa.

The self-hosting break-even calculation

The worked math below lands the crossover around $4,000–5,000/month of API spend at mid-2026 pricing — call it $4,200/month, or roughly 5–7 billion tokens/month depending on your input-to-output ratio. That number is derived from the illustrative assumptions in this article, not an industry constant. Below the threshold, the fully-loaded cost of self-hosting — including idle GPU time, ML engineer overhead, and tooling — exceeds what you pay the API provider.

The calculation is worth doing explicitly for your situation rather than trusting a generic number:

MANAGED API COST
  Input tokens:  volume_in  × price_in  (e.g., $0.40/M for GPT-4.1-class)
  Output tokens: volume_out × price_out (e.g., $1.60/M for GPT-4.1-class)
  Total = monthly API bill

SELF-HOSTED COST (Llama 3.3 70B INT8 served with vLLM on 1× H100 80GB)
  The model choice IS the quality-parity assumption — a break-even against
  GPT-4-class pricing only holds if the open model passes your eval suite.
  GPU instance:    $2.50–3.50/hr on spot, $4.00–6.00/hr reserved
  Per month (730h): $1,825–4,380 depending on tier and commitment
  Throughput:      ~2,000–4,000 tok/s sustained (prefill + decode combined) at 70% utilization
                   Decode is memory-bandwidth-bound at 70B; the 8–12k tok/s
                   figures you'll see quoted are for 7–8B models.
  Engineer overhead: 0.05–0.10 FTE × $200k loaded = $833–1,667/month
  Tooling/infra:   ~$300–500/month (monitoring, object storage, networking)
  Total self-hosted ≈ GPU + engineer + tooling

BREAK-EVEN: managed_cost = self_hosted_cost
  At $0.40/M input, $1.60/M output, 70/30 input/output split:
  → ~$4,200/month or ~5–7B tokens/month (at this blended rate) is the practical crossover
  Note: "11B tokens" figures you'll see elsewhere assume ~$0.40/M all-in or older pricing.
  
CRITICAL INPUTS TO GET RIGHT:
  - GPU utilization rate (60% is realistic; 40% makes self-hosting ~25% more expensive)
  - Engineer hourly rate (a $350k principal costs 75% more than a $200k mid-level)
  - Idle time cost (GPUs still cost money when traffic drops at 3 AM)

The number that surprises most teams is engineer overhead. A $200,000 fully-loaded annual ML engineer cost is $16,667/month. Even at 5% of their time on inference infrastructure — which is optimistic if something breaks — that is $833/month of human cost before touching a single GPU. At 10% time, it exceeds the GPU cost at low volumes entirely. And that overhead is concrete work, not abstraction: someone operates vLLM or SGLang (or TensorRT-LLM if you need every last token per second), owns model version upgrades and canary rollouts, tunes batching parameters, and gets paged when the server OOMs on a long-context request at 2 AM.

The practical implication: treat $4,200/month as a trigger condition to evaluate self-hosting, not as an automatic switch. Write the threshold in an Architecture Decision Record before you start building — "When monthly API spend exceeds $X for three consecutive months, revisit the infrastructure decision" — and revisit it with actual data when the trigger fires.

Data residency changes the calculus completely

There is one scenario where the cost math doesn't matter: regulated data. But be precise about what actually forbids external APIs, because most compliance requirements don't. HIPAA permits PHI on a managed API under a Business Associate Agreement — Azure OpenAI, Bedrock, and Anthropic all offer HIPAA-eligible endpoints, and a BAA plus in-region processing is the standard compliant path for healthcare workloads, not self-hosting. SOC 2 is an attestation about your controls, not a regulation that bans external processors. The genuinely hard cases are contractual no-third-party-processor terms, sovereign-cloud mandates, and air-gapped environments. There, self-hosting in a compliant environment is the only path, regardless of whether the volume justifies it economically.

This is the one case where you pay the self-hosting premium at any scale. The good news is that small open-weight models (Llama 3.x, Qwen 2.5, Mistral) now perform well enough on most document processing and extraction tasks that you do not need to self-host a frontier-scale model to meet quality requirements.

Small vs large: the actual decision criteria

The small-vs-large question has a clear answer for most tasks — test the small model first. The pattern that recurs in production systems is that teams default to the largest available model for every call because it "handles everything," then discover months later that a 7B model fine-tuned on their domain outperforms the frontier model on their specific task and costs 30x less.

Before you get excited about that 30x: the fine-tuned model is not free, and its cost is exactly the "build" side of build-vs-buy. Count it:

THE FINE-TUNE PATH, FULLY COSTED (illustrative, mid-2026)
  Labeled data:      2,000–5,000 examples for a narrow extraction/classification task
                     ~$0.50–2.00/example annotated in-house or via vendor → $1,000–10,000 one-time
                     (Cheaper if you can distill labels from a frontier model — validate a sample by hand)
  LoRA training:     a few hundred dollars of GPU compute per run (hours on 1–8× A100/H100)
  Eval harness:      1–2 engineer-weeks to build; you need it regardless of model choice
  Re-training cadence: domain drift forces a refresh every 1–2 quarters —
                     budget the labeling + training cost as recurring, not one-time

At high volume the math still works — a one-time $5–15k investment against a 10–30x per-token saving pays back in weeks. At low volume it doesn't: below a few hundred dollars a month of spend on the task, the fine-tune never recoups its data cost. Run the payback calculation before the training job.

Here is the honest breakdown of where size matters:

Task type	Small model (1–7B)	Large model (>30B)
Classification / intent routing	Excellent — a fine-tuned 0.5B BERT achieves ~90% routing accuracy in <10ms	Overkill; adds latency and cost with no quality benefit
Named entity extraction	Good for common entities; misses rare schemas	Needed for novel schemas and complex nested structures
Structured JSON extraction	Good with output constraints (grammar-based decoding)	Better on ambiguous or multi-step extractions
Multi-step reasoning	Poor — degrades sharply on reasoning chains >3 steps	Strong — the main remaining advantage
Long-document synthesis	Poor on >8K context; attention quality degrades	Better through ~128K context, though still degrades
Code generation (common patterns)	Good — StarCoder2 3B, DeepSeek-Coder 7B perform well	Needed for novel architecture, cross-file context
Open-ended generation	Noticeably worse on fluency and edge cases	Meaningfully better on creative and complex tasks

The practical engineering recommendation: run both models on 200 samples from your actual production distribution before committing. The automated eval suite you need for this is cheap to build — see why eval is hard for the measurement framework, and model routing and cascades for how to operationalize the routing once you have the data.

{ "type": "model-router", "title": "Interactive routing: cost vs quality at different thresholds" }

The open-weight quality gap closed

In 2023, the choice between managed APIs and open-weight models was a real quality trade-off — GPT-4 was substantially better. By mid-2026, Llama 3.3 70B, Qwen 2.5 72B, Mistral Large, and DeepSeek-V3 match or beat GPT-4-class quality on coding, instruction following, and structured extraction benchmarks. The remaining gaps are specific:

• Long-horizon reasoning: o1/o3-class models with test-time compute scaling still lead significantly
• Multimodal: closed models (GPT-4o, Gemini 2.x) lead on vision and audio tasks
• Reliability on rare edge cases: closed frontier models handle unusual prompts more gracefully
• Tool use in complex agent workflows: closed models still show better reliability at >5 tool-call depth

For most production classification, extraction, and summarization tasks, open-weight models are now the economically correct choice if you're already self-hosting for other reasons.

Speculative decoding: bridging quality and speed

The one technique that changes the small-vs-large latency trade-off meaningfully is speculative decoding. The mechanism: a small draft model generates k candidate tokens in parallel; the large target model verifies them all in a single forward pass; accepted tokens advance the output, rejected tokens get resampled from the target's distribution. The target model output is identical in distribution to what it would have produced alone.

sequenceDiagram
    participant D as Draft Model (3B)
    participant T as Target Model (70B)
    participant O as Output

    D->>T: propose ["The", "answer", "is", "Paris", ","] (5 tokens)
    T->>T: verify all 5 in single forward pass
    T-->>O: accept ["The", "answer", "is", "Paris"] — reject ","
    T-->>O: resample next token from target distribution
    Note over D,T: Net result: 4 tokens at cost of ~1 target forward pass

The speedup depends on acceptance rate — the fraction of draft tokens the target accepts. On predictable outputs (structured JSON, code with constrained schemas, domain-specific factual answers), acceptance rates reach 70–90% and EAGLE3 delivers 3–5x speedup. On open-ended generation, acceptance rates drop to 40–60% and speedup falls to 1.5–2x.

This matters for self-hosting decisions: if your primary concern is latency on a large model, speculative decoding may make self-hosting viable for latency-sensitive paths where a managed API's round-trip adds 200–400ms of network overhead. It does not help on managed APIs because you cannot control the serving infrastructure.

Quantization: the memory math

Quantization is the mechanism that makes small-model serving practical on affordable GPUs, and large-model serving practical on a smaller H100 fleet. The numbers:

VRAM REQUIREMENTS (approximate, mid-2026)
  Llama 3.3 70B in bf16:    ~140 GB → requires 2× H100 80GB minimum
  Llama 3.3 70B in INT8:    ~70 GB  → fits on 1× H100 80GB
  Llama 3.3 70B in INT4:    ~35 GB  → fits on A100 40GB with headroom
  Qwen 2.5 7B in bf16:      ~14 GB  → fits on RTX 4090 24GB with tight headroom
  Qwen 2.5 7B in INT4:      ~3.5 GB → fits on a consumer GPU

ACCURACY IMPACT (general benchmarks)
  bf16 → INT8:  < 1% degradation on most tasks
  INT8 → INT4:  1–5% degradation depending on method (GPTQ vs AWQ)
  INT4 (AWQ):   Better preservation than GPTQ on reasoning tasks
  TurboQuant (3-bit KV cache, March 2026): 6× KV cache compression, ~0% accuracy loss

The practical hierarchy: start with INT8 for any safety-critical or reasoning-heavy task. Use INT4 when you have validated it on your specific domain and task — the 1–5% average accuracy degradation hides 10–15% degradation on specific hard examples that may be precisely the ones that matter for your application. TurboQuant's 6x KV cache compression is significant because KV cache is often the binding memory constraint at high context lengths, not the model weights — it enables longer contexts on the same GPU without any model quality change.

flowchart LR
    TASK["Production task"] --> SAFE{"Safety-critical\nor reasoning-heavy?"}
    SAFE -->|Yes| INT8["INT8 quantization\n~2x memory reduction\n<1% accuracy loss"]
    SAFE -->|No| EVAL{"Validate INT4 on\nyour eval suite"}
    EVAL -->|Pass threshold| INT4["INT4 (AWQ/GPTQ)\n~4x memory reduction\n1–5% accuracy loss"]
    EVAL -->|Fail threshold| INT8
    INT8 --> KV{"High context\nlengths (>32K)?"}
    INT4 --> KV
    KV -->|Yes| TURBO["TurboQuant KV cache\n6x compression\nnear-zero quality impact"]
    KV -->|No| DONE["Proceed to serving"]
    style INT8 fill:#15803d,color:#fff
    style INT4 fill:#0e7490,color:#fff
    style TURBO fill:#a855f7,color:#fff

The infrastructure decision tree in full

Combining both axes into a single decision process:

flowchart TD
    START["New LLM workload"] --> MEAS["Instrument cost attribution first\n(gateway tags: feature, model, user)"]
    MEAS --> SCALE{"Monthly API spend?"}
    SCALE -->|"< $4,200/month"| OPT["Optimize first:\n1. Caching (prompt-cache-prefix-semantic-llm)\n2. Routing to small models\n3. Batch API for async workloads"]
    SCALE -->|"> $4,200/month 3+ months"| PRIV{"Data residency\nrequirement?"}
    PRIV -->|Yes| SELFHOST_REQ["Self-host required\nSelect open-weight model\nfor compliance"]
    PRIV -->|No| UTIL{"GPU utilization\n> 60%?"}
    UTIL -->|No| FIX["Fix utilization first\n(batching, traffic shaping)"]
    UTIL -->|Yes| SELFHOST["Self-host viable\nRun 3-month cost pilot"]
    OPT --> QUALITY{"Task needs frontier\nquality?"}
    QUALITY -->|No| SMALL["Use small model\n(1–7B)\nTest on eval suite first"]
    QUALITY -->|Yes| LARGE["Use frontier model\nEval-confirm the need"]
    style MEAS fill:#ffaa00,color:#0a0a0f
    style SMALL fill:#15803d,color:#fff
    style SELFHOST fill:#0e7490,color:#fff
    style SELFHOST_REQ fill:#ff2e88,color:#fff

What breaks

The premature self-hosting trap. A team at $1,500/month of API spend decides to self-host to "control costs." They spend three months building the serving infrastructure, burn 0.3 FTE of a senior engineer, and end up with a system that costs $2,800/month all-in (GPU + engineer time) compared to the $1,500/month they were paying. The invisible cost is what didn't get built during those three months. The rule: do not self-host until you have hit the cost threshold three consecutive months and have verified your GPU utilization plan.

GPU utilization collapse at night. Self-hosted inference costs money 24/7. A feature with predictable business-hours traffic might show 80% GPU utilization at 2 PM and 5% at 2 AM. The average utilization determines whether the economics work. Autoscaling GPU instances helps but adds cold-start latency (minutes for a new GPU node to come online). Spot instances are cheaper but interrupted arbitrarily. Teams that do not model their traffic distribution before committing to self-hosting almost always overpay.

Model size chosen by default, not eval. The most common form of waste: a team routes every call to the same frontier model because it was the model they used in the prototype and they never revisited the decision. Six months in, 60% of calls are simple intent classification queries that a 500M-parameter BERT classifier would handle with better latency and 1/50th the cost. The fix is easy: build the eval suite, run both models, route based on data. See model routing and cascades for the implementation.

Quantization degradation on domain-specific edge cases. Average benchmark accuracy for INT4 models is 95–99% of bf16 quality. But averages obscure the distribution. A financial document extraction task that depends on correctly parsing rare entity formats may see 15–20% degradation at INT4 precisely on the hard examples that matter most for compliance. Always validate quantization against your specific domain eval suite, not just general benchmarks.

The context window mismatch in fallback chains. If you build a two-tier system where the primary model is a 128K-context frontier model and the fallback is a self-hosted 8K-context small model, a long prompt that exceeds 8K tokens will either silently truncate or fail on the fallback. This is a class of bug that is invisible until a specific user hits it. Every fallback chain needs explicit context-window-aware prompt degradation — truncation, summarization, or refusal — before invoking the smaller model. The fallbacks, retries, and circuit breakers article covers the implementation.

Routing without observability. A router that silently downgrades hard queries to a weak model because the confidence score was above threshold produces a failure mode that looks like model quality degradation with no clear signal. You get user complaints, you revert the router, you learn nothing. Every routing decision must be logged — query text or embedding, routing decision, model used, confidence score, output quality score from your eval pipeline. Without that, you cannot close the feedback loop. See the LLM gateway pattern for the instrumentation layer.

The managed API market in 2026

As of mid-2026, the managed API market has four tiers worth knowing:

Tier	Examples	Price range (input)	When to use
Frontier closed	GPT-4.1, Claude Sonnet 4, Gemini 2.x	$0.40–3.00/M tokens	Complex reasoning, multimodal, edge-case sensitivity
Frontier open via API	Llama 3.3 via Together/Groq, Qwen 2.5 via Alibaba	$0.10–0.40/M tokens	GPT-4-class quality tasks, latency-tolerant, cost-sensitive
Small model API	Gemini Flash ($0.10/M), GPT-4o-mini ($0.15/M), Mistral-Small	$0.10–0.20/M tokens	High-volume classification, extraction, routing
Managed dedicated	Bedrock provisioned throughput, Azure PTUs, Fireworks/Baseten/Together dedicated endpoints	Flat $/GPU-hr or $/unit (illustrative: $2–10/hr per instance)	Steady predictable volume; open-weight economics without the self-hosting ops burden

The competitive pressure in 2024–2026 collapsed frontier prices to a few percent of 2023 levels: GPT-4-class quality that cost $30/M input tokens in early 2023 now costs ~$0.40/M — a ~75x drop. This changes the self-hosting calculus: the absolute dollar cost threshold rises as API prices fall. At $0.10/M input tokens (Gemini Flash), the self-hosting break-even shifts to roughly 40B tokens/month or ~$4,000/month — roughly comparable, but the quality bar for what you get at that price is now substantially lower than it was.

{ "type": "token-cost", "title": "Monthly bill by model tier, caching, and volume" }

Batch API: the underused lever

For workloads without a real-time SLA — document processing, classification pipelines, embedding generation, overnight summarization — managed batch APIs cut costs by approximately 50% with no quality trade-off. Both OpenAI and Anthropic offer batch endpoints where you submit a JSONL file and receive results within 24 hours. At $4,000/month of non-interactive API spend, switching to batch saves $2,000/month with a single configuration change.

# Anthropic batch API — simple, cheap, identical quality
import anthropic

client = anthropic.Anthropic()

# Build a batch of requests from your pipeline
requests = [
    {
        "custom_id": f"doc-{i}",
        "params": {
            "model": "claude-sonnet-4-5",
            "max_tokens": 1024,
            "messages": [{"role": "user", "content": doc_text}],
        },
    }
    for i, doc_text in enumerate(documents)
]

# Submit the batch
batch = client.messages.batches.create(requests=requests)
print(f"Batch ID: {batch.id}, status: {batch.processing_status}")

# Poll for completion (or use webhooks in production)
import time
while True:
    result = client.messages.batches.retrieve(batch.id)
    if result.processing_status == "ended":
        break
    time.sleep(60)

# Stream results
for item in client.messages.batches.results(batch.id):
    if item.result.type == "succeeded":
        print(f"{item.custom_id}: {item.result.message.content[0].text}")

The one place batch API breaks: if you are processing documents that users are waiting for in a web UI. Even with perceived-latency tricks (progress bars, streaming partial results from a cache), a 24-hour SLA is not a web experience. The right split is to identify which workloads are genuinely async — jobs that run on a schedule, reports generated overnight, embeddings for documents uploaded to a knowledge base — and move those to batch while keeping the interactive path on real-time endpoints.

The judgment call

The framework collapses to four questions asked in order:

First: do you have cost attribution? If not, all other decisions are guesswork. Stand up a gateway with per-feature, per-model tagging before optimizing anything. The LLM gateway pattern covers the implementation; LiteLLM and Portkey (Apache 2.0 since March 2026) both provide this out of the box.

Second: are you above the self-hosting threshold? If your API spend has exceeded $4,200/month for three consecutive months, run the full self-hosting analysis with your actual traffic distribution and utilization projections — plugging in your real GPU-hours, idle rate, and engineer allocation, not a generic rule of thumb. If you are below it, do not self-host. Spend the engineering time on caching and routing instead — prompt caching deep dive and token economics cover the 5–10x bill reduction that is available without any infrastructure work.

Third: what does your eval suite say about model size? Default to the smallest model that passes your quality threshold. Test explicitly. Do not assume. A fine-tuned 7B model on domain-specific data usually outperforms a general-purpose 70B model on that domain's specific tasks — and costs 10x less to serve.

Fourth: are there compliance requirements? Data residency needs establish a self-hosting requirement regardless of cost. In that case, start with small open-weight models (Llama 3.x, Qwen 2.5, Mistral) at INT8 precision, validate quantization against your domain eval suite, and add speculative decoding if you have latency pressure and the serving infrastructure to support a draft model.

The most expensive mistake in LLM infrastructure is not picking the wrong model or the wrong host — it is making the decision before you have the data to make it correctly.