Token Economics: Where Your API Bill Actually Comes From

A startup I know spent three months assuming their LLM cost problem was about picking a cheaper model. They switched from GPT-4 to Sonnet, saved maybe 30%, and still had an uncomfortable bill. When they finally added per-feature cost logging, they found a single internal admin tool — used by five employees — making 8,000-token requests because someone had forgotten to remove the full document corpus from the system prompt. It ran 200 times a day. The model cost was irrelevant. The context was the problem.

This is the first thing to internalize about LLM economics: the model tier matters, but context bloat usually matters more.

Why output costs more than input

Every LLM request runs in two distinct phases. The prefill phase processes your input tokens — the system prompt, conversation history, any retrieved context, and the user's message. This is embarrassingly parallel: all input tokens are processed simultaneously in a single forward pass. It is compute-bound — the GPU reads the model weights once and amortizes that read across every input token, so the arithmetic units stay saturated. Fast and relatively cheap.

The decode phase generates your output tokens. This is autoregressive: to produce token N, the model must have already produced tokens 1 through N-1. Each token requires a full forward pass. If you request a 500-token completion, you get 500 sequential forward passes. This is memory-bandwidth-bound, not compute-bound — the GPU must re-read the full model weights for every single generated token — and it does not parallelize across tokens in a single request.

The pricing reflects this. As of mid-2026 (illustrative — check provider pages, these move):

Provider / Model	Input ($/M tokens)	Output ($/M tokens)	Output multiplier
GPT-4o (OpenAI)	$2.50	$10.00	4×
Claude Sonnet 4 (Anthropic)	$3.00	$15.00	5×
Gemini 2.0 Flash (Google)	$0.10	$0.40	4×
GPT-4o mini (OpenAI)	$0.15	$0.60	4×
DeepSeek V3 (API)	$0.27	$1.10	~4×

The multiplier is consistent across providers because the underlying hardware economics are the same. Budget accordingly: if your application generates 500-token responses for 200-token inputs, that is 2.5× the tokens at 4× the price — your output cost runs ~10× your input cost before any context overhead.

The practical implication: applications where users need long, detailed responses (document drafting, code generation, analysis) pay much more per call than applications where short structured outputs suffice (classification, intent detection, yes/no checks). A 50-token classification output is 10× cheaper to generate than a 500-token summary, at the same input length.

The context overhead problem

Here is the math that surprises most people. Take a production chatbot with a 1,500-token system prompt and a 100-token user message at the start of a session. That first call costs 1,600 input tokens. Reasonable.

sequenceDiagram
    participant U as User
    participant A as App
    participant API as LLM API

    U->>A: Turn 1 (100 tokens)
    A->>API: system(1,500) + user(100) = 1,600 input tokens
    API-->>A: assistant(300 tokens)

    U->>A: Turn 5 (100 tokens)
    A->>API: system(1,500) + history(4×400) + user(100) = 3,200 input tokens
    API-->>A: assistant(300 tokens)

    U->>A: Turn 15 (100 tokens)
    A->>API: system(1,500) + history(14×400) + user(100) = 7,200 input tokens
    API-->>A: assistant(300 tokens)

If each turn includes 300 tokens of assistant output that gets appended to history, by turn 15 the conversation history alone is 14 × 400 tokens = 5,600 tokens per call. The user's message is still 100 tokens. You are paying for 72× more context than content.

By turn 30, input tokens per call in a naive implementation are often 5–10× what they were at turn 1. In a high-volume chatbot this compounds into a severe cost surprise. A session that costs $0.005 at turn 1 costs $0.025–0.05 at turn 20. If users have long sessions, your per-session cost scales with session length, not query length.

The fix is a rolling window with summarization: keep the last N turns verbatim, summarize everything older into a compact context block. An 800-token summary can compress 20 turns of 8,000 tokens of history. This caps per-call input growth while preserving enough context for coherent responses. You pay one extra API call to generate the summary when the window overflows. It is almost always worth it.

from anthropic import Anthropic

client = Anthropic()

def compress_history(history: list[dict], max_turns: int = 10) -> list[dict]:
    """Keep last max_turns; summarize everything older."""
    if len(history) <= max_turns:
        return history

    older = history[:-max_turns]
    recent = history[-max_turns:]

    # One summarization call — cheap, short output
    summary_response = client.messages.create(
        model="claude-haiku-4-5",  # cheapest model for compression
        max_tokens=300,
        messages=[
            {
                "role": "user",
                "content": (
                    "Summarize this conversation history in 200 words, "
                    "preserving key facts and decisions:\n\n"
                    + "\n".join(f"{m['role']}: {m['content']}" for m in older)
                ),
            }
        ],
    )

    summary_block = {
        "role": "user",
        "content": f"[Earlier conversation summary]: {summary_response.content[0].text}",
    }
    return [summary_block] + recent

The system prompt tax

Every API call ships the system prompt as input tokens. This is not a per-session cost; it is a per-call cost. A 2,000-token system prompt at 10,000 calls/day burns 20M input tokens every single day.

At $1/M input (illustrative, mid-tier model), that is $20/day or $600/month on system prompt alone, before a single user message is counted. Expand to a larger deployment — 100,000 calls/day — and system prompt becomes $6,000/month of pure overhead.

The obvious advice is to trim the system prompt, and that is correct. Audit it periodically. A common finding: instructions added for edge cases that never actually occur in production, tool schemas for tools that were removed months ago, example conversations that made sense during development but add tokens in production. A methodical trim from 3,000 tokens to 800 tokens is not unusual for systems that grew organically.

The less obvious solution is prefix caching. Both Anthropic and OpenAI support it. When the leading part of your prompt is identical across calls, the provider caches the KV cache state for that prefix and charges you a fraction of the full input price on subsequent calls — roughly 10% of list price for Anthropic (cache reads at $0.30/M vs $3.00/M fresh), and a 50% discount via OpenAI's automatic caching. For a 2,000-token system prompt sent 100,000 times per day, prefix caching is the single highest-ROI optimization available. More on the mechanics in the Prompt Caching deep dive.

The structural requirement: put the stable, cacheable content first. System instructions before user content, static RAG documents before the query, few-shot examples before the live request. Anything that changes per call — the user message, a timestamp, a dynamic tool result — must come at the end, after the stable prefix. Shuffle this order and the cache misses.

{ "type": "prompt-cache", "scenario": "chatbot", "title": "Prompt anatomy: cached vs cache-busting segments" }

Output length: your biggest cost lever

For applications where you control the output, max output tokens is a direct cost dial.

A 1,000-token response costs 5–10× more to generate than a 100-token response, at identical input length. If your use case is classification, summarization into a fixed schema, or yes/no decisions with brief justification, hard-enforce short outputs. Set max_tokens explicitly. Instruct the model to be concise in the system prompt. Test whether output quality degrades — for structured tasks it usually does not.

# Classification: 50-token output budget, structured response
response = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=50,  # hard cap — classification does not need 500 tokens
    messages=[{"role": "user", "content": f"Classify this support ticket: {ticket}"}],
    system=(
        "Classify the input into exactly one of: [billing, technical, account, other]. "
        "Respond with JSON: {\"category\": \"...\", \"confidence\": 0.0-1.0}. "
        "No explanation."
    ),
)

For generative tasks where length is user-driven (chat, document drafting), you cannot hard-cap without hurting experience. The optimization here shifts to routing: send that long-form generation task to a cheaper model where quality is comparable. More on this in Model Routing and Cascades.

Reasoning tokens: output you pay for but never see

Reasoning-tier models (OpenAI's o-series and GPT-5 thinking modes, Claude with extended thinking, Gemini thinking variants) add a line item that ambushes naive estimates: thinking tokens. The model generates internal chain-of-thought before the visible answer, and every provider bills those tokens at the output rate — even when the reasoning text is never returned in the response. The ratio is not small. A 300-token visible answer backed by 3,000 thinking tokens bills 3,300 output tokens; at $5/M output that is $0.0165 per call instead of the $0.0015 your visible-output estimate predicted — 11x. Multiply across a feature that quietly got switched to a reasoning model and your dashboard looks broken before anyone remembers the config change.

Two controls. First, the effort or budget knob — reasoning_effort on OpenAI, a thinking token budget on Anthropic (names drift; the mechanism is a cap on deliberation) — should be set deliberately per feature, not left at the default. Most classification and extraction tasks need no reasoning at all. Second, your instrumentation must log reasoning tokens as their own field, separate from visible output tokens; otherwise per-feature output averages become meaningless the moment one route uses a thinking model.

A related pricing wrinkle: long-context tiers. Some providers charge a higher $/M rate once a request crosses a context threshold (for example, above 200k input tokens). A RAG pipeline that occasionally overstuffs the window pays the premium rate on the whole request, not just the tokens over the line — one more reason the context-bloat fixes above come before model shopping.

Model tier: real numbers, honest comparison

GPT-4-level quality that cost $30/M input tokens in early 2023 now costs $0.40–$1.00/M — roughly 30–75x cheaper, a price drop of over 95%. This collapse is important context: at 2023 prices, model tier was the dominant cost variable. At 2026 prices, waste patterns matter more than model selection for most teams.

That said, the tier spread is still real:

Relative cost (illustrative, mid-2026, normalized to "frontier input = 1×"):

                    Input    Output
Frontier tier       1.0×     1.0×      (GPT-4o, Claude Opus, Gemini Pro)
Mid tier            0.15×    0.15×     (GPT-4o mini, Claude Haiku, Gemini Flash)
Cheap/small tier    0.03×    0.03×     (DeepSeek distills, Llama-3.1-8B via API)

At 1B input tokens / month:
  Frontier:  $1,000–$3,000
  Mid:         $150–$450
  Small:        $30–$90

The 10–50× cost spread between frontier and small models is meaningful. For tasks where small models perform acceptably — classification, extraction, rewriting, short-form Q&A on narrow domains — routing them to the cheaper tier is not a quality-versus-cost trade-off. It is free cost reduction. The mistake is assuming that every call needs the frontier model when you have not actually tested the alternatives.

The test protocol is simple: take a sample of 200 real production inputs for the feature you are evaluating. Run both models. Compare outputs on your quality rubric (human or LLM-as-judge). If the cheaper model achieves >90% of the quality score at 10% of the cost, the routing decision is easy.

Batch API: the 50% discount most teams ignore

For workloads that are not user-facing — document summarization pipelines, classification jobs, embedding generation, nightly report runs — the async Batch API from OpenAI and Anthropic cuts costs 50% with no change to the model or quality.

The trade-off is latency: batch jobs complete within 24 hours rather than in real time. For background jobs that is irrelevant. The error is treating every API call as real-time when the downstream consumer is a database write or an email that goes out tomorrow.

# OpenAI Batch API: submit a file of requests, get results back in bulk
import json
from openai import OpenAI

client = OpenAI()

# Build JSONL batch file
requests = [
    {
        "custom_id": f"classify-{i}",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
            "model": "gpt-4o-mini",
            "messages": [
                {"role": "system", "content": "Classify sentiment: positive/negative/neutral. JSON only."},
                {"role": "user", "content": review},
            ],
            "max_tokens": 20,
        },
    }
    for i, review in enumerate(reviews)
]

with open("batch_input.jsonl", "w") as f:
    for req in requests:
        f.write(json.dumps(req) + "\n")

# Upload and submit
batch_file = client.files.create(file=open("batch_input.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
# Poll later: client.batches.retrieve(batch.id)

The 50% discount is applied automatically when you use the Batch API endpoint. No configuration, no separate contract. If you are running any nightly or async pipeline through the standard chat completions API, you are paying double for no reason.

The interactive cost visualizer

{ "type": "token-cost", "title": "Monthly bill builder: adjust your workload" }

Drag the sliders to model your actual workload. The output-premium callout in the widget makes the asymmetry between input and output pricing tangible.

What breaks

Context accumulation that nobody catches

Conversation history grows silently. In development you test short sessions; in production users have 50-turn conversations. The cost curve looks linear in staging and exponential in production. The fix is not to cap session length — it is to instrument average input tokens per call as a monitored metric and alert when it trends upward. If you do not have this metric, you will not see the problem until the monthly bill lands.

Embedding calls counted as zero cost

Embedding API calls are cheap per token — often $0.02–0.13/M tokens. They look negligible. But a RAG pipeline that re-embeds every document on every query, or a semantic cache that embeds every user message, can accumulate hundreds of millions of embedding tokens per day. These do not show up in "LLM API cost" dashboards unless you explicitly attribute them. Separate your embedding spend from your completion spend in your accounting.

Token counting estimated wrong

Many teams estimate costs using len(text.split()) * 1.33 as a rough token count. This breaks on code (high token density), URLs, numbers (each digit may be a separate token), and multilingual text (non-ASCII languages often tokenize at lower efficiency, producing more tokens per character). The right approach is to use the tokenizer directly.

import tiktoken  # for OpenAI models
import anthropic  # count_tokens for Anthropic

# OpenAI
enc = tiktoken.encoding_for_model("gpt-4o")
token_count = len(enc.encode(text))

# Anthropic — synchronous count, no generation
client = anthropic.Anthropic()
count = client.messages.count_tokens(
    model="claude-opus-4-5",
    messages=[{"role": "user", "content": text}],
)
print(count.input_tokens)

Using actual token counts instead of estimates typically reveals costs 10–40% higher than naive word-count approximations on real production data.

Output length spikes from adversarial prompts

Users who ask the model to "write a comprehensive 5,000-word report on X" are generating output tokens you did not budget for. Without a hard max_tokens ceiling at the application layer, individual requests can spike to 10x the expected output length. At 4–5× the cost of input tokens, a handful of runaway responses can distort your daily cost metrics significantly. Always set max_tokens. If users legitimately need long outputs, consider charging for them separately or implementing per-user output budgets via the LLM Gateway pattern.

The measurement prerequisite

None of the optimizations above are useful if you cannot see their effect. The minimum viable cost instrumentation is four fields logged on every API call:

import time
from dataclasses import dataclass

@dataclass
class LLMCallRecord:
    timestamp: float
    model: str
    feature: str          # e.g., "customer-support-chat", "document-summarizer"
    user_id: str | None
    input_tokens: int
    output_tokens: int
    cached_input_tokens: int  # 0 if caching not enabled
    latency_ms: float
    estimated_cost_usd: float

def log_llm_call(response, feature: str, user_id: str | None = None) -> LLMCallRecord:
    usage = response.usage
    # Illustrative pricing — parameterise from config in production
    INPUT_COST_PER_M = 3.00
    CACHE_COST_PER_M = 0.30
    OUTPUT_COST_PER_M = 15.00

    # Anthropic semantics: usage.input_tokens already EXCLUDES cache reads
    # and cache writes — they are reported in separate fields. (OpenAI is
    # the opposite: cached_tokens is a subset of prompt_tokens, so there
    # you would subtract. Do not mix the two conventions.)
    fresh_input = usage.input_tokens
    cached = getattr(usage, "cache_read_input_tokens", 0) or 0
    cache_writes = getattr(usage, "cache_creation_input_tokens", 0) or 0

    cost = (
        fresh_input * INPUT_COST_PER_M / 1_000_000
        + cached * CACHE_COST_PER_M / 1_000_000
        + cache_writes * INPUT_COST_PER_M * 1.25 / 1_000_000  # writes bill at 1.25x
        + usage.output_tokens * OUTPUT_COST_PER_M / 1_000_000
    )

    record = LLMCallRecord(
        timestamp=time.time(),
        model=response.model,
        feature=feature,
        user_id=user_id,
        input_tokens=usage.input_tokens,
        output_tokens=usage.output_tokens,
        cached_input_tokens=cached,
        latency_ms=0,  # fill from actual timing
        estimated_cost_usd=cost,
    )
    # Ship to your observability store here
    return record

Without per-feature attribution, "we spend $15k/month on LLM APIs" tells you nothing actionable. With it, you discover that two features are responsible for 60% of the spend, one of which could route to a cheaper model without any quality impact.

This is the actual prerequisite for everything in the Cost Control and Reliability Engineering curriculum. The optimization techniques — caching, routing, batching, output length control — only move the needle when you can see the before and after.

How your bill actually compounds in production

Let me make the compounding concrete with a realistic production scenario. You are building an AI writing assistant. One feature is "improve my draft" — the user pastes a document, the model rewrites it. Documents average 1,500 tokens. The system prompt is 1,800 tokens. Output averages 1,400 tokens. You run 5,000 calls/day.

Per-call cost (mid-tier model, illustrative at $1/M input, $5/M output):
  Input:  (1,800 system + 1,500 doc) = 3,300 tokens = $0.0033
  Output: 1,400 tokens = $0.0070
  Total/call: $0.0103

Daily: 5,000 × $0.0103 = $51.50
Monthly: ~$1,545

Now add prefix caching to the 1,800-token system prompt (90% discount on cached hits at 95% hit rate):

Without caching:
  System prompt cost/call: 1,800 × $1/M = $0.0018

With prefix caching (95% hit rate, cache at $0.10/M):
  0.95 × 1,800 × $0.10/M = $0.000171
  0.05 × 1,800 × $1.00/M = $0.000090
  Effective system prompt cost/call: $0.000261

Savings per call: $0.0018 - $0.000261 = $0.001539
Monthly saving: 5,000 × 30 × $0.001539 = $230.85/month

That is $230/month saved by restructuring your prompt order to enable prefix caching and enabling one API flag. No change to model, no change to prompt content, no change to output quality. This is why understanding token economics precedes every other optimization conversation.

The decision framework

Before reaching for any specific technique — caching, routing, batching, compression — answer these questions in order:

flowchart TD
    START["New LLM feature or cost spike"] --> MEASURE{"Do you have per-feature\ncost breakdown?"}
    MEASURE -->|No| INSTRUMENT["Add model, feature, input/output token\nlogging first. Come back in a week."]
    MEASURE -->|Yes| ASYNC{"Is the workload\nasync / non-interactive?"}
    ASYNC -->|Yes| BATCH["Use Batch API (50% discount)\nbefore any other optimization"]
    ASYNC -->|No| CTXBLOAT{"Is context > 3× the\nuseful content?"}
    CTXBLOAT -->|Yes| TRIM["Trim system prompt, add rolling\nwindow history, enable prefix caching"]
    CTXBLOAT -->|No| OUTLEN{"Is output > 200 tokens for\na structured task?"}
    OUTLEN -->|Yes| MAXTOKEN["Add max_tokens constraint\nand output format instruction"]
    OUTLEN -->|No| ROUTE{"Is this task actually\nfrontier-model-hard?"}
    ROUTE -->|No| SMALLMODEL["Route to cheaper model tier\n(test first on 200 real samples)"]
    ROUTE -->|Yes| OK["You're optimized. Ship it."]
    style INSTRUMENT fill:#ff2e88,color:#111
    style BATCH fill:#15803d,color:#fff
    style TRIM fill:#15803d,color:#fff
    style MAXTOKEN fill:#15803d,color:#fff
    style SMALLMODEL fill:#15803d,color:#fff
    style OK fill:#0e7490,color:#fff

The order matters. Measurement before optimization is not a platitude — without per-feature data, you will optimize the wrong thing. Batch API before prompt tuning because it is the least invasive 50% saving you can find. Context before routing because bloated context inflates cost regardless of model tier.

When model tier actually matters

Model tier becomes the primary cost lever when:

1. You have already eliminated context bloat (overhead tokens < 2× useful tokens per call).
2. You have enabled prefix caching on static prompt segments.
3. You have separated async workloads to the Batch API.
4. Your per-feature breakdown shows one feature still expensive despite steps 1–3.

At that point, test routing that feature to a cheaper model. For the model routing mechanics and confidence-based cascades, the routing article covers them in depth. The point here is sequencing: most teams reach for routing before eliminating context bloat. The context bloat fix is faster, simpler, and often saves more.

The pricing collapse since 2023 has not made token economics irrelevant — it has changed where the real savings are. At $30/M in 2023, model choice was everything. At $0.40–1.00/M in mid-2026, you can afford to be on the right model for the job while the system prompt tax and context bloat remain the dominant cost variables. Master those, and the rest of this section — caching, routing, batching, reliability — becomes additive improvement on a sensible base.