Tokens, Costs, and the Context Window

The feature shipped on a Tuesday. The team celebrated. On Friday someone forwarded the AWS Cost Explorer screenshot to the engineering channel, and then there was a very quiet weekend.

The feature was fine. The $40,000 invoice was not.

What happened was not a bug in the usual sense. Nobody forgot a semicolon. The team had shipped a customer support chatbot that passed the full conversation history — all prior turns, verbatim — plus a 2,000-token system prompt on every single message. At turn 15 of a long support conversation, each API call was sending 12,000 input tokens. At 50,000 conversations per day averaging 20 turns each, they were burning through roughly eight billion tokens a day. That is a lot of $0.003/thousand invoices.

Token costs are not intuitive. They compound in ways that surprise even experienced engineers. This module is the math and mental model you need to never be surprised by an invoice again.

Text does not go into the API. Tokens do.

Before anything else: models do not see your text. They see a sequence of integers, each representing a chunk of text determined by a Byte-Pair Encoding (BPE) vocabulary trained on the pretraining corpus.

BPE starts from individual bytes or characters and greedily merges the most frequent adjacent pair, building up a vocabulary of subword chunks. The result is a learned artifact of training data distribution: common English words like "the", "and", "function" get their own token, while rare words split into pieces.

{ "type": "tokenizer", "title": "How your text becomes token IDs" }

A few numbers that will save you estimation mistakes:

• English prose: ~1.3 tokens per word (a token is ~0.75 words), or ~1,300 tokens per 1,000 words
• Code: heavier than prose — brackets, underscores, and indentation all fragment into extra tokens, and JSON structures are especially expensive
• Arabic, Thai, Korean: 2-3x more tokens per word than English equivalents
• Numbers: 1,234,567 often splits into multiple tokens, which is why LLMs are bad at arithmetic

The API charges per token, not per word or per character. Estimate in tokens, not words.

The pricing asymmetry that dominates every bill

Every major LLM API charges separately for input tokens and output tokens, and output tokens cost more. Not a little more — typically 3-5x more.

Why? Processing your input is the prefill phase: the GPU ingests all your input tokens in parallel in a single forward pass — high arithmetic intensity, compute-bound, the kind of work GPUs are built for. Generating output is the decode phase: each new token requires a full forward pass attending to every prior token, one token at a time, and every step streams the entire set of model weights plus the KV cache through GPU memory just to emit a single token — memory-bandwidth-bound, with terrible hardware utilization per token. You cannot parallelize it. Generation is expensive by nature.

As of mid-2026, illustrative prices for a frontier-class model look roughly like:

Input:  $1.00 – $3.00 per million tokens
Output: $4.00 – $15.00 per million tokens

Gemini Flash sits closer to $0.10/M input; GPT-4-level models cluster around $2.50/M input. Prices drift, so anchor on the mechanism (the ~5x output premium) rather than the exact numbers, and check your provider's pricing page before doing projections.

The implication: a response that's twice as long as it needs to be costs twice as much in output tokens. "Be concise" is not just UX advice. It's a billing directive.

The context window: your model's RAM

Here is the mental model that will stick: the context window is RAM. Your program's working memory. Everything the model can know right now lives inside it. Everything outside does not exist from the model's perspective.

{ "type": "context-window", "window": 128000, "title": "Context window as a fixed budget" }

The window is a hard limit on total tokens — input plus output combined — for one API call. Exceed it and the API returns an error. Unlike RAM, you cannot swap to disk; information that does not fit simply cannot be seen.

What fills a context window?

flowchart LR
    SP[System prompt] --> W[Window budget]
    TC[Tool definitions] --> W
    RC[Retrieved chunks] --> W
    CH[Chat history] --> W
    UQ[User query] --> W
    OR[Output reserve] --> W
    style W fill:#a855f7,stroke:#a855f7,color:#fff

In a multi-turn chat application, the chat history grows with every exchange. By turn 15, if you're passing the full conversation verbatim, you've spent far more tokens on history than on the current question. This is how the $40,000 invoice happens.

As of mid-2026, frontier model window sizes range from 128K (GPT-4o class) to 1M+ (Gemini 1.5/2.0 Pro, Claude 4.x). But advertised context size and effective context size are not the same thing. Research repeatedly shows that retrieval accuracy degrades at the middle of long contexts — the "lost in the middle" effect. Models tend to weight tokens at the start and end of the context more heavily than those buried in the middle. Fill a 128K window with 120K tokens of background documents and the model may not reliably retrieve something buried at the 60K mark.

For production systems, this means: more context is not always better. A well-indexed RAG pipeline retrieving the right 2K tokens often outperforms stuffing 50K tokens of "maybe relevant" content into context, both on quality and cost. The full treatment is in RAG in One Pass.

Why long context is expensive: KV cache intuition

You need a mental model of the KV cache to understand why long context carries a cost beyond just counting input tokens.

When the model processes your prompt, it computes key (K) and value (V) tensors for every token in every layer. These tensors encode "what this token knows about itself and its neighbors." During generation, the model needs them to attend over prior context. The KV cache stores them so they are not recomputed for every new output token.

The problem: KV cache memory scales with context length. Specifically:

KV cache size ≈ 2 × num_layers × num_kv_heads × head_dim × context_length × dtype_bytes

For a 70B model with Grouped-Query Attention (GQA) at 128K context:

2 × 80 layers × 8 KV heads × 128 dim × 128,000 tokens × 2 bytes (fp16)
≈ 42 GB just for the KV cache

That's 42 GB of GPU memory before you run a single forward pass for generation, and before you account for the 140 GB of model weights. This is why you cannot just throw every frontier model onto a consumer GPU and serve 100K-token contexts — the memory is consumed by the cache.

{ "type": "kv-cache", "model": "70b", "context": 65536, "gpu": "a100-80", "title": "KV cache dominates GPU memory at long context" }

The practical consequence for API users: providers price long-context calls higher (or throttle them) because they are genuinely more expensive to serve. Long contexts also have higher TTFT because the prefill phase is proportional to input length. A 10K-token prompt takes ~10x longer to prefill than a 1K-token prompt. If TTFT matters to your users, prompt length is the main lever.

The worked math: a chatbot at 100k requests/day

Let's run the numbers for a real production scenario. Assume a customer support chatbot:

• Traffic: 100,000 requests/day
• Prompt anatomy: 500-token system prompt + 200-token average conversation history + 100-token user message = 800 tokens input
• Response: average 300 tokens output
• Model: mid-tier frontier ($1.50/M input, $6.00/M output, illustrative)
• Scale: 30 days

# Scenario A: no optimization
Input:  100k × 30 × 800 tokens  = 2.4B tokens/month
Output: 100k × 30 × 300 tokens  = 0.9B tokens/month

Input cost:  2,400M × $1.50/M  = $3,600
Output cost:    900M × $6.00/M  = $5,400
                                  -------
Monthly total:                    $9,000

Now let's add prompt caching. The 500-token system prompt is identical on every request and can be cached.

# Scenario B: cache the system prompt
Cached input: 100k × 30 × 500 tokens = 1.5B tokens, at $0.15/M (cache read ~10% of fresh)
Fresh input:  100k × 30 × 300 tokens = 0.9B tokens, at $1.50/M

Cached cost:  1,500M × $0.15/M = $225
Fresh cost:     900M × $1.50/M = $1,350
Output cost:    900M × $6.00/M = $5,400
                                  ------
Monthly total:                    $6,975

That is a $2,025/month reduction by changing nothing except where you put the system prompt and enabling caching. No quality change. No model change.

Two honest caveats before you copy these numbers. Scenario B assumes every request is a cache hit — realistic at 100k requests/day against a hot cache, wildly optimistic for low-traffic or bursty apps where cached prefixes expire between requests. And providers enforce a minimum cacheable prefix length: a 500-token system prompt on its own is below Anthropic's ~1,024-token floor and would not cache at all there — you'd bundle it with tool definitions or few-shot examples to clear the bar. More on both in the next section.

Now route 60% of traffic (simple queries) to a small model at $0.10/M input, $0.30/M output:

# Scenario C: routing + caching
Simple (60%): 60k req/day × 30 × 800 tokens input × $0.10/M = $144
              60k req/day × 30 × 300 tokens output × $0.30/M = $162
Frontier (40%): using Scenario B math, scaled to 40% = $6,975 × 0.4 = $2,790
                                                                        ------
Monthly total:                                                           $3,096

The $9,000 bill drops to $3,096 — about a 3x reduction — with routing plus caching. This is not a trick. These are real mechanisms available today. The full playbook is in Cost Control and Reliability Engineering.

Prompt caching: the highest-leverage lever

Prompt caching works by reusing the KV tensors the provider already computed for a stable prefix. When two requests share an identical leading segment — same system prompt, same tools list, same retrieved documents — the provider detects the match and skips the prefill computation for that segment.

The key constraint: the cache key is an exact prefix match. If anything changes at the beginning of the prompt, the cache is busted for everything that follows.

sequenceDiagram
    participant App
    participant Provider
    participant KVStore
    App->>Provider: Request 1 [system prompt][user msg]
    Provider->>KVStore: Store KV tensors for system prompt
    Provider-->>App: Response (full cost)
    App->>Provider: Request 2 [system prompt][different user msg]
    Provider->>KVStore: Cache hit on system prompt prefix
    KVStore-->>Provider: Cached KV tensors
    Provider-->>App: Response (90% cheaper for cached segment)

{ "type": "prompt-cache", "scenario": "chatbot", "title": "Which prompt segments survive caching" }

Three things that bust your cache without you noticing:

1. A timestamp in the system prompt. "Today is July 2, 2026, 14:32:07 UTC" changes every request.
2. Shuffled tool definitions. If you programmatically construct your tools list and the order varies, the prefix changes.
3. Growing history at the wrong position. Cache-stable segments must come first; conversation history (which grows) must come after. If your code prepends the user message and appends the system prompt, you've inverted this.

The structural rule: put stable content first, dynamic content last.

# Good: cache-stable prefix first
messages = [
    {"role": "system", "content": STATIC_SYSTEM_PROMPT},   # cached
    *STATIC_FEW_SHOT_EXAMPLES,                               # cached
    *conversation_history,                                   # dynamic, comes after
    {"role": "user", "content": user_message},               # dynamic, last
]

# Bad: dynamic content at the top busts the cache
messages = [
    {"role": "system", "content": f"Today is {datetime.now()}. {STATIC_SYSTEM_PROMPT}"},
    ...
]

With Anthropic's API, you signal which prefix to cache explicitly:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": STATIC_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},  # mark this for caching
        }
    ],
    messages=conversation_history + [{"role": "user", "content": user_message}],
)

# Check cache usage in the response
print(response.usage.cache_read_input_tokens)    # tokens served from cache
print(response.usage.cache_creation_input_tokens) # tokens written to cache this call

OpenAI's automatic caching requires no code changes — it caches any prompt prefix of 1,024+ tokens automatically and applies a 50% discount. The tradeoff: you don't control what gets cached or see granular hit/miss data.

Three constraints bite in production, and the marketing pages bury all of them. First, cache writes cost extra on Anthropic: the first request that populates the cache pays ~1.25x the base input price for the cached segment, so a prefix that gets written but never re-read is a net loss. Second, cached prefixes expire: Anthropic's ephemeral cache has a ~5-minute TTL (refreshed on each hit), so an endpoint that gets one request every ten minutes sees a near-0% hit rate no matter how perfect the prompt structure is. Third, minimum prefix lengths apply — roughly 1,024–2,048 tokens depending on model. As of mid-2026, the comparison looks like:

	Anthropic (explicit)	OpenAI (automatic)
Cache read	~$0.30/M vs $3/M fresh (90% off)	50% off
Cache write	~1.25x base input price	No premium
TTL	~5 min ephemeral, refreshed on hit	Minutes to ~1 hour, provider-managed
Minimum prefix	~1,024–2,048 tokens (model-dependent)	1,024 tokens

The deeper article is at Prompt Caching: Cutting Costs 50-90% Without Changing Model Behavior.

Streaming and latency basics

Streaming and cost are often confused. Let me separate them.

Streaming (Server-Sent Events) means the API sends tokens to your client as they are generated, rather than buffering the complete response and sending it at once. This does not reduce the number of tokens generated or the total compute. It reduces perceived latency — the user sees the first word arrive in ~500ms rather than waiting 8 seconds for the whole answer.

Two latency metrics that matter:

• TTFT (Time to First Token): how long from sending the request to receiving the first response token. Dominated by prefill time, which scales with input length. Long prompts → high TTFT.
• Tokens per second (throughput): how fast the model generates output. Depends on the model, hardware, and batch load at the provider. You cannot control this from the client side.

sequenceDiagram
    participant C as Client
    participant P as Provider
    C->>P: Send request (800 tokens in)
    Note over C,P: TTFT clock starts at send
    Note over P: Prefill phase: process 800 tokens in parallel
    P-->>C: Token 1
    Note over C,P: TTFT ends when Token 1 arrives
    P-->>C: Token 2
    P-->>C: Token 3 ...
    Note over P: Decode phase: one token at a time
    P-->>C: Final token + usage stats

For interactive applications: optimize TTFT by shortening prompts (and by caching — cache hits dramatically reduce prefill time since the cached segment is served from memory, not recomputed). For background pipelines, throughput matters more than TTFT.

One trap: streaming retries. If a streaming response fails halfway through, naive retry logic sends the user a new response from the beginning — they see duplicate or contradictory content. Either handle partial failure gracefully (detect mid-stream errors and surface them cleanly) or don't stream for workloads where retry correctness matters. The full treatment is in Streaming LLM Responses.

What breaks in practice

Context length ≠ context quality. Stuffing a 100K-token prompt does not guarantee the model uses it well. The "lost in the middle" effect is documented across multiple models: information buried in the center of a long context is recalled less reliably than information at the start or end. If you're relying on a specific fact being retrieved from 60K tokens in, measure whether it actually is. Often a RAG approach that retrieves the relevant 2-4K tokens outperforms the full-context approach on both accuracy and cost.

The caching math fails when structure breaks. Teams often instrument caching, see a 0% hit rate, and conclude caching does not work. Usually the prompt structure is wrong: a dynamic timestamp or user ID is embedded in the system prompt, and every request looks like a cache miss. Audit your prompts with token-level logging before concluding caching has no ROI.

Output token cost surprises. Engineers budget based on the prompt, not the completion. Then a model returns a 2,000-token chain-of-thought preamble before the actual answer. Set max_tokens explicitly on every call. For structured outputs where you know the approximate response size, this bounds your worst-case cost. For open-ended generation, monitor output length distributions in production — they shift with model updates and prompt changes.

Multi-turn history accumulation. A 20-turn conversation with no history management means the final request carries 19 prior turns. If each turn averages 400 tokens, the final call has 7,600 tokens of history plus the current message. Typical approaches: sliding window (keep last N turns), summary compression (LLM-summarize older history), or selective retention (keep only turns flagged as relevant). The tradeoff between context coherence and cost is real and context-dependent — there is no universal right answer, but there is a universal wrong one: no management at all.

Non-English cost estimation. If your application serves Arabic, Thai, Hebrew, Korean, or Japanese users, your token estimates from English testing will be off by 2-3x. Budget and size context windows accordingly. The Tokenization and BPE article goes deeper into why.

Where to go next

The module you just read covers the economics. The adjacent topics that extend it:

• Context Engineering: Filling the Window Right — what to put in the window and in what order, with specific techniques for managing multi-turn history without blowing the budget.
• Prompt Caching: Cutting Costs 50-90% Without Changing Model Behavior — the full prefix caching and semantic caching treatment, including implementation patterns for OpenAI and Anthropic, cache invalidation strategies, and the 2025 security research on adversarial cache collisions.
• Token Economics: Where Your API Bill Actually Comes From — the full cost breakdown including batch API, context-window bloat in multi-turn sessions, and cost attribution per feature.
• Context Windows, KV Cache, and the Reality Behind 1M Token Claims — the serving-side math, GQA's role in shrinking KV cache footprint, and why 1M-token windows perform inconsistently in practice.
• Cost Control and Reliability Engineering — model routing, cascades, fallbacks, and how to build a system that degrades gracefully when the primary provider is down.
• RAG in One Pass: Build, Measure, Improve — for when context stuffing hits its limits and you need retrieval to do the heavy lifting.