From Prototype to Production: Scaling an AI Product

The demo worked perfectly. Every question in the stakeholder meeting got a sharp, accurate answer. You shipped it. Three weeks later the monthly API bill is $40,000, your ops team is asking why a support bot is occasionally telling customers their subscription renews in 2018, and the product manager wants to know why responses went from two seconds to eight seconds "after that prompt change on Thursday." Nothing crashed. The system is technically healthy. That is the production AI problem.

Why prototypes lie about scale

A prototype runs at tiny traffic. The cost is invisible. The quality issues that exist at 1,000 queries per month get lost in the noise. Latency that is "acceptable" at 10 concurrent users becomes a p99 disaster at 1,000. And most critically: the prototype has no mechanism to tell you when it gets worse.

The economics are the starkest signal. Frontier model pricing as of mid-2026 runs roughly $5-20 per million tokens depending on the provider and tier. At 1,000 queries per day with ~2,000 tokens average, that is ~$30/day. At 100,000 queries per day — ten weeks of user growth for a successful product — the same math gives $3,000/day without any architectural change. That is the trap. Nothing in the codebase changes. The bill just grows.

The LLM gateway pattern is the first structural fix, and it should be added before you hit scale, not in response to it. A gateway gives your application a single entry point to all LLM providers, with model aliases that decouple your code from specific model IDs. When you write model="chat-standard" instead of model="claude-opus-4-5-20251001", swapping providers becomes a config change, not a code change across 14 services. LiteLLM is the leading open-source implementation; Cloudflare AI Gateway, Kong AI Gateway, and Portkey are managed alternatives.

The cost stack: where the money actually goes

Before you can cut costs intelligently, you need to know what you are paying for. Token economics have three components: input tokens, output tokens, and the prompt caching tier.

Per-request cost anatomy (illustrative — high-end frontier model, mid-2026):
  Input tokens (system prompt + context + user message): 1,500 tokens × $15/M  = $0.0225
  Output tokens (response):                               500 tokens × $60/M   = $0.0300
  Total per request: ~$0.053

At 100k requests/day: ~$5,300/day = ~$159,000/month

With prompt caching on a 1,200-token static system prompt
(cache hits at ~90% rate, cached price ~10% of input price):
  Uncached, the prefix costs: 1,200 × $15/M × 100,000                  = $1,800/day
  Cached: 90% of reads at $1.50/M + 10% misses at $15/M = $162 + $180  = $342/day
  Savings: ~$1,458/day — a ~65% cut to this workload's total input spend ($2,250/day),
  because the static prefix dominates the input. Workloads with short system prompts see less.

Output tokens cost 3-5x more than input tokens at most providers — a fact that surprises most teams because input is where you feel like you're spending. The implication is that output verbosity is expensive. max_tokens is not just a timeout; it is a cost control. Every unnecessary word in a response has a price.

The token economics breakdown covers this in full. The key production insight is that you need per-query cost attribution from day one — not just total spend. Without per-query cost you cannot tell which features are economical and which are destroying the margin.

Model routing: the 85% lever

The most impactful cost intervention at scale is routing queries by complexity. Not all queries need a frontier model. A customer asking "what are your business hours?" does not need the same reasoning capacity as one asking "explain the tax implications of my last five transactions." Sending them to the same model is waste.

A routing layer sits between your application and the model tier selection. The simplest implementation is a cheap fast classifier — either a small language model, a fine-tuned embedding classifier, or even a rule-based heuristic on query length and vocabulary — that labels each incoming query as simple, medium, or complex.

import litellm

ROUTING_RULES = {
    "simple": "gpt-4o-mini",        # ~$0.15/M input — customer lookup, FAQ, format conversion
    "medium": "claude-sonnet-4-5",  # ~$3/M input — reasoning, multi-step, domain knowledge
    "complex": "claude-opus-4-5",   # ~$5/M input — legal analysis, code architecture, novel reasoning
}

def classify_complexity(query: str) -> str:
    """Fast heuristic classifier — replace with a fine-tuned model at scale."""
    words = set(query.lower().split())
    word_count = len(query.split())
    # Word-boundary match, not substring — 'if' must not fire on 'difference'
    has_conditional = bool(words & {"if", "unless", "compare", "analyze", "why"})
    if word_count < 20 and not has_conditional:
        return "simple"
    elif word_count < 100 or not has_conditional:
        return "medium"
    return "complex"

def routed_completion(query: str, system_prompt: str) -> str:
    tier = classify_complexity(query)
    model = ROUTING_RULES[tier]
    response = litellm.completion(
        model=model,
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": query},
        ],
    )
    return response.choices[0].message.content

The production version of this classifier is itself a fine-tuned model trained on your query distribution, with confidence thresholds that escalate uncertain queries to the next tier. The documented result at scale is 85% cost reduction — achievable when the simple-query bucket is large, which it typically is in customer-facing applications where the Pareto principle applies hard: 80% of queries are variations on 20% of intents.

The model routing and cascades article goes deeper on cascade architectures where the output of the cheap model gates whether the expensive model is called at all.

{ "type": "model-router", "title": "Routing queries by complexity: live cost vs quality meters" }

Prompt caching: highest ROI, most skipped

Prompt caching is the optimization that pays for itself immediately and requires zero change to model behavior. All three major providers support it as of mid-2026, with different mechanics: OpenAI caches repeated prefixes automatically (~50% discount, no code change), Anthropic uses explicit cache_control breakpoints (~90% discount on cache reads), and Google Gemini offers context caching with managed cache objects. The mechanism is the same everywhere: the provider stores the KV cache of a processed prompt prefix, and subsequent requests that share that prefix pay a fraction of the normal input token cost — roughly 10% on Anthropic, ~50% on OpenAI — for the cached portion.

The conditions for a good cache hit:

1. The prefix must be identical byte-for-byte (no dynamic content in the cached region).
2. The prefix must be long enough to be worth caching — Anthropic's minimum is 1,024 tokens.
3. The same prefix must appear frequently enough to amortize the initial processing cost.

System prompts with long policy documents, tool definitions, or RAG context that applies to all queries are ideal. The antipatterns that destroy cache efficiency:

# BAD: timestamp in the system prompt busts the cache on every call
system_prompt = f"""
You are a helpful assistant. Current time: {datetime.now().isoformat()}.
[... 2,000 more tokens of policy ...]
"""

# GOOD: stable prefix; inject time-sensitive content in the user turn
SYSTEM_PROMPT_CACHED = """
You are a helpful assistant for Acme Corp customer support.
[... 2,000 tokens of policy, tool definitions, examples ...]
"""

def build_messages(user_query: str) -> list:
    return [
        {"role": "system", "content": SYSTEM_PROMPT_CACHED},
        {"role": "user", "content": f"[{datetime.now().strftime('%H:%M')}] {user_query}"},
    ]

The prompt caching deep dive covers prefix caching versus semantic caching (handled at the gateway layer, not the provider layer). Semantic caching intercepts queries that are semantically equivalent to a cached query and returns the cached response — useful for FAQ-style traffic but requires careful staleness handling.

Context engineering: the 2026 shift

By mid-2026, the term "prompt engineering" had narrowed to a specific and somewhat limited practice, while the broader discipline of designing what a model sees on each call got its own name: context engineering. The distinction matters operationally.

Prompt engineering asks: how should I phrase this instruction? Context engineering asks: given a token budget of N, what is the optimal mix of system instructions, retrieved documents, tool definitions, conversation history, and in-context examples to include on this specific call?

The shift was driven by agent failures. When an agent fails at a multi-step task, the root cause is almost never the phrasing of the system prompt. It is almost always one of:

• The relevant fact was not in context (retrieval failure or budget eviction).
• A stale or irrelevant tool definition consumed space that should have gone to context.
• The conversation history grew to a size that triggered context rot.
• The model's attention was diffused across too many weakly relevant chunks.

Context rot is a documented production phenomenon: model quality measurably degrades as context window size grows, even on tasks the model handles perfectly with smaller context. This is not intuitive — more context should help. But empirically, on complex reasoning tasks, very large contexts often hurt. Context size needs to be a monitored metric in production, not just a headroom number.

flowchart LR
    Q[User query] --> CTX[Context assembler]
    CTX --> JIT["Just-in-time retrieval\n(top-k relevant docs only)"]
    CTX --> MASK["Tool masking\n(only tools relevant to intent)"]
    CTX --> COMP["History compaction\n(summarize turns > N)"]
    JIT --> WIN[Context window budget]
    MASK --> WIN
    COMP --> WIN
    WIN --> |"Fits in budget?"| OK[Model call]
    WIN --> |"Overflow"| EVICT["Evict by priority:\nhistory, tools, retrieved, core"]
    EVICT --> OK

    style CTX fill:#00e5ff,color:#0a0a0f
    style WIN fill:#a855f7,color:#fff
    style EVICT fill:#ff2e88,color:#111

Just-in-time retrieval is the most impactful context engineering technique: instead of including all potentially relevant documents up front, retrieve only the top-k most relevant to the current query, letting relevance scores drive the selection. The context window management article covers the full budget allocation strategy.

Tool masking is underrated. If your agent has 30 available tools but the current query is about scheduling, including all 30 tool definitions wastes tokens on distractor noise. A fast intent classifier that selects the 5-8 most relevant tools for each query meaningfully improves both quality and cost.

Async batching and latency architecture

Real-time streaming is not always the right model. For workloads where users can wait — document analysis, report generation, bulk classification — async batching changes the economics significantly.

Continuous batching (the technique vLLM, SGLang, and TensorRT-LLM use for self-hosted inference) dynamically groups requests into GPU computation batches, achieving roughly 3x throughput improvement versus static batching. For cloud API calls you do not control batching directly, but you can control your request scheduling: queue non-interactive work, tag it by priority tier, and dispatch the low-priority queue through a batch endpoint in scheduled windows instead of firing synchronous calls as requests arrive. The batch endpoints are the headline lever here — OpenAI's Batch API and Anthropic's Message Batches API (GA since December 2024) both price at ~50% of synchronous rates in exchange for a completion window of up to 24 hours. A nightly bulk-classification job that would cost $400/day as synchronous calls costs ~$200/day batched, with identical outputs.

The streaming, batching, and throughput tradeoffs article covers the mechanics. For the production decision framework:

Workload type	Latency requirement	Best approach
Interactive chat	< 500ms TTFT (time to first token)	Streaming, warm connection pool
Search-augmented Q&A	< 2s total	Streaming with retrieval prefetch
Document analysis	Best-effort, minutes OK	Async queue, batch dispatch
Bulk classification	Throughput-optimized	Batch API endpoints (OpenAI Batch, Anthropic Message Batches) at ~50% of sync price
Scheduled reports	No latency SLA	Nightly batch at off-peak pricing

Serverless inference (Lambda-style per-invocation billing) handles bursty workloads without idle GPU cost, but cold-start latency — which can reach 5-30 seconds for large model loads — kills interactive SLAs. The production pattern is a warm pool: a small number of always-on instances that handle steady-state traffic, with serverless overflow for spikes. The warm pool size is the optimization variable.

What breaks

The prompt change regression

The leading cause of unexpected production regressions in 2025-2026 is a prompt change with no evaluation gate. A developer rewords the system prompt to fix one edge case. It ships. Three days later, a different class of queries — one that wasn't in the manual test cases — starts producing worse answers. HTTP 200 on every request. No alarm. The regression surfaces in user complaints two weeks later.

The fix is prompt CI/CD: version every prompt artifact in source control, define baseline evaluation metrics, and gate deployment on those metrics. A prompt cannot ship if evaluation scores fall below baseline thresholds. The prompt versioning and CI/CD article covers the pipeline in detail. The critical principle: prompts are versioned, tested artifacts, not prose.

Provider dependency failure

Your application calls openai.chat.completions.create(model="gpt-5") directly. OpenAI sunsets that model version. Or they raise prices. Or they have a regional outage. Every one of these becomes an incident that requires a code change across all services that hardcoded the model ID.

The gateway abstraction solves this: your code calls model="chat-standard", the gateway resolves the alias, and you swap the underlying provider with a config change. Fallback chains in the gateway (primary: claude-opus-4-5, fallback: gpt-4o) handle outages without application-layer changes. The reliability fallbacks article covers circuit breaker patterns for the full failure taxonomy.

Context accumulation and session blowup

Agents that maintain conversation history have a structural failure mode: the context grows without bound as sessions lengthen. A session that starts within a 32k token budget might hit 200k tokens after an hour of interaction. This has two effects: cost increases linearly with session length, and quality degrades because of context rot.

The production solution is staged context compaction: when the history exceeds a threshold, summarize earlier turns and replace them with the summary. The summary token count is a fraction of the original. Quality drops slightly on old-history recall but stays sharp on recent context.

MAX_HISTORY_TOKENS = 8_000  # threshold before compaction kicks in
SUMMARY_PREFIX = "[Earlier conversation summary]"

def cheap_model_complete(prompt: str) -> str:
    # Summarization goes to a cheap small model — same litellm call as the routing snippet
    response = litellm.completion(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content

def compact_history(messages: list, current_token_count: int) -> list:
    if current_token_count < MAX_HISTORY_TOKENS:
        return messages

    # Keep the real system prompt + last 4 turns. Everything else — including
    # the summary injected by a previous compaction pass — gets folded into the
    # new summary, so summaries replace each other instead of stacking up.
    system = [m for m in messages
              if m["role"] == "system" and not m["content"].startswith(SUMMARY_PREFIX)]
    recent = messages[-8:]   # last 4 user/assistant pairs
    older = messages[len(system):-8]  # prior summary + turns between system and recent

    if not older:
        return messages

    summary_prompt = "Summarize this conversation history in ≤300 tokens, preserving key facts and decisions:\n"
    summary_prompt += "\n".join(f"{m['role']}: {m['content'][:500]}" for m in older)

    summary = cheap_model_complete(summary_prompt)
    summary_message = {"role": "system", "content": f"{SUMMARY_PREFIX}: {summary}"}

    return system + [summary_message] + recent

The silent quality cliff

Infrastructure metrics (latency p50/p99, error rate, token count) are necessary but miss the failures that matter. A system can achieve 99.9% uptime while silently producing wrong answers on 15% of queries. This is the defining failure mode of production AI systems: HTTP 200 does not mean the answer was correct.

Output quality sampling is non-negotiable. The production pattern: sample 1-5% of queries, run them through an LLM-as-judge evaluator (a separate model call that scores the response on faithfulness, relevance, and task completion), and push the scores to your observability dashboard. The LLM observability article covers the full signal stack.

sequenceDiagram
    participant U as User
    participant APP as Application
    participant LLM as LLM Provider
    participant OBS as Observability

    U->>APP: query
    APP->>LLM: completion request
    LLM-->>APP: response (HTTP 200)
    APP-->>U: response shown

    Note over APP,OBS: async, 1-5% sampling
    APP->>OBS: log (query, response, metadata)
    OBS->>LLM: LLM-as-judge quality score request
    LLM-->>OBS: faithfulness: 0.87, relevance: 0.92
    OBS->>OBS: alert if score < threshold or drift detected

The behavioral drift signal is equally important: if the distribution of response lengths, refusal rates, or topic clusters shifts unexpectedly, something changed — a model version update, a context accumulation pattern, or a prompt regression. Statistical process control applied to response distribution is the right framing.

Guardrails belong in code, not prompts

Adding a sentence to the system prompt ("never discuss competitor products") is not a guardrail. It is a hope. Under adversarial input, prompt-based restrictions fail at rates that increase with prompt complexity. Production guardrails belong in code:

• Input guardrails: validate and sanitize before the model sees the query. Classify intent, detect injection attempts, enforce schema on structured inputs.
• Output guardrails: parse and validate model output before returning it to users. Schema enforcement, PII redaction, content policy classifiers.
• Agent circuit breakers: in agentic systems, halt execution if cost per turn exceeds threshold, if the iteration count exceeds a limit, or if tool call patterns suggest a loop. These belong in the agent loop code, not the system prompt.

The guardrails in production article covers the NeMo Guardrails and Llama Guard implementation patterns.

The data flywheel: making production work for you

The most expensive thing about production AI is paying frontier model prices forever. The economic escape route is the data flywheel: capture production failures, curate them into training signal, fine-tune a smaller model on that signal, and gradually route high-volume tasks to the cheaper specialist.

NVIDIA's published case study is instructive: their employee support agent achieved 94-96% accuracy at significantly lower per-request cost after running the data flywheel for several cycles. The small fine-tuned model replaced frontier calls for the majority of support queries. The frontier model stayed in the loop only for the novel edge cases.

The flywheel inputs:

• Implicit signals: thumbs-down ratings, session abandonment, follow-up "that's wrong" queries, human agent corrections of AI outputs. These are 10-100x more abundant than explicit ratings and require instrumentation at the application layer to capture.
• Explicit signals: ratings, corrections, approval workflows. Lower volume, higher signal quality.
• Failure cases: queries where the current model returned a low quality score, triggered a guardrail, or required human intervention.

The agent-in-the-loop pattern automates the curation step: human agents correcting AI outputs generate correction pairs automatically, without a separate annotation workflow. The corrections become preference data for DPO or examples for SFT.

The data flywheel article covers the full curation-to-training pipeline. The key production reality: data quality matters far more than data volume. A curated set of 10,000 high-quality examples with difficulty weighting and coverage balancing outperforms 100,000 randomly sampled production logs.

The decision in practice: when to invest in each layer

The scaling investments described here are not all needed on day one. The sequencing is:

At 1,000 queries/day: Add the AI gateway with model aliases and unified retry logic. Set up per-query cost attribution. Enable prompt prefix caching if your system prompt is large. This costs almost nothing operationally and removes the most painful future retrofits.

At 10,000 queries/day: Add model routing for the simple-query bucket. Add quality sampling (1% of traffic through LLM-as-judge). Version your prompts in source control with basic eval gates.

At 100,000 queries/day: The math in the back-of-envelope section becomes urgent. Add semantic caching at the gateway. Implement context compaction for agent sessions. Invest seriously in the eval pipeline — prompt versioning CI/CD is table stakes at this scale, and A/B testing for any significant prompt or model change is standard practice.

At 1,000,000 queries/day: The frontier model is almost certainly not your primary cost driver for most query types by now — you have either routed them away or fine-tuned a specialist. The data flywheel should be collecting signal continuously. Self-hosted inference for the fine-tuned models starts to pencil out against API costs at sufficient volume.

The pattern across all these thresholds: the reference architecture stays the same; what changes is which layers carry real load. A production AI system is not architecturally different from a prototype — it is a prototype with every implicit assumption made explicit and every shortcut replaced with something that holds under real conditions.

The honest summary of what separates the AI products that survive their first year from those that do not: the ones that survive had a gateway before they needed one, versioned their prompts before they had a regression, added quality monitoring before a user complaint surfaced the problem, and started capturing training signal before the flywheel felt urgent. That is the prototype-to-production gap. It is entirely engineering, not magic, and it is all solvable.