Prompt Caching: Cutting Costs 50-90% Without Changing Model Behavior

Last October, a team at a Series B company found that their new AI coding assistant was burning $47,000/month on LLM API calls — roughly double the forecast. The feature had shipped cleanly, users liked it, and latency was fine. The problem: every request sent the complete 4,200-token tool schema and system prompt through full prefill, every time, even though neither had changed in three months. One afternoon of prompt restructuring and adding cache_control markers brought that to $14,000/month. The model's behavior didn't change by a single token.

That's the whole pitch for prompt caching. It's not a trick or a workaround — it's reusing computation the provider already did. Understanding the mechanism tells you exactly how to exploit it.

How KV cache reuse actually works

When an LLM processes input tokens, it computes key and value matrices for each token at each attention layer — the KV cache. For a fresh request, this computation (the prefill phase) runs for every input token before the model generates the first output token. Prefill is compute-bound and proportional to the number of input tokens. A 4,000-token input takes roughly 4× as long to prefill as a 1,000-token input, and costs proportionally more.

The insight: if today's request starts with the same 3,000 tokens as yesterday's 10,000 requests, the provider computed those KV states 10,000 times. Prompt caching stores those states (on GPU or fast storage) and reuses them when the prefix matches.

sequenceDiagram
    participant C as Client
    participant P as Provider API
    participant KV as KV Cache Storage

    Note over C,KV: First request — cache miss, full prefill
    C->>P: [system: 3000 tokens][user: 200 tokens]
    P->>P: Prefill all 3200 tokens (full cost)
    P->>KV: Store KV states for prefix (3000 tokens)
    P-->>C: Response + cache_creation_input_tokens: 3000

    Note over C,KV: Second request (same prefix) — cache hit
    C->>P: [system: 3000 tokens][user: 400 tokens]
    P->>KV: Retrieve cached KV states (prefix matches)
    P->>P: Prefill only new 400 tokens
    P-->>C: Response + cache_read_input_tokens: 3000

The savings come from skipping the prefill compute for the cached portion. This is why cache hits also reduce time-to-first-token: the prefill phase for 3,000 tokens takes real wall-clock time that's simply not paid on a hit.

One clarification on model behavior: the output is identical to a fresh prefill of the same tokens. Providers guarantee this — the KV states are mathematically equivalent to running prefill fresh. If you're seeing behavioral differences after enabling caching, something else changed (model version, temperature, the actual token sequence).

Provider mechanics side by side

Anthropic: explicit markers, maximum savings

Anthropic's implementation requires you to tell it exactly where the cacheable prefix ends by placing {"type": "text", "text": "...", "cache_control": {"type": "ephemeral"}} markers in the content. The last marker in the sequence defines the cache boundary.

import anthropic

client = anthropic.Anthropic()

# System prompt with cache marker — this entire block gets cached
system = [
    {
        "type": "text",
        "text": STATIC_SYSTEM_PROMPT,  # Role, constraints, output format — never changes
        "cache_control": {"type": "ephemeral"},
    }
]

# Tool schemas with their own cache marker (if very long)
# tools = [...with cache_control on the last tool...]

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    system=system,
    messages=[
        {
            "role": "user",
            "content": user_message,  # Dynamic — never cached
        }
    ],
)

# Check what actually got cached
usage = response.usage
print(f"Input tokens:          {usage.input_tokens}")
print(f"Cache creation tokens: {usage.cache_creation_input_tokens}")
print(f"Cache read tokens:     {usage.cache_read_input_tokens}")

On a cache miss, cache_creation_input_tokens shows the tokens that were prefilled and stored. On a hit, cache_read_input_tokens shows the tokens that were served from cache. You pay 1.25× the base input price for cache_creation_input_tokens — the write premium — and 10% for cache_read_input_tokens.

The cache TTL is 5 minutes, refreshed on each cache hit. A high-traffic endpoint that hits the cache dozens of times per minute keeps it warm indefinitely — there's no ceiling, just the requirement that hits keep arriving inside the window. A separate 1-hour TTL option exists for gappier traffic, at a 2× write cost. An endpoint that gets one request every 6 minutes will always miss the 5-minute cache.

OpenAI: automatic, no markers

OpenAI caches the longest matching prefix automatically, no markers or configuration needed. The minimum cacheable prefix is 1,024 tokens. Cache hits appear in the response usage as cached_tokens:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": STATIC_SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

usage = response.usage
print(f"Total input tokens:  {usage.prompt_tokens}")
print(f"Cached tokens:       {usage.prompt_tokens_details.cached_tokens}")
print(f"Uncached tokens:     {usage.prompt_tokens - usage.prompt_tokens_details.cached_tokens}")

The discount is 50% on cached tokens, versus Anthropic's 90%. OpenAI's cache TTL is roughly 5–10 minutes of inactivity — shorter and less predictable than Anthropic's refreshable 5-minute window. There's no explicit way to force cache warming; the first request always pays full price.

The tradeoff: OpenAI's zero-configuration approach is easier to start with, but Anthropic's explicit markers give you more control — you can cache multiple checkpoints within a single prompt and you get predictable accounting.

{"type": "prompt-cache", "scenario": "chatbot", "title": "Cache-stable vs cache-busting prompt segments"}

The structural rule: static before dynamic

Every caching strategy reduces to one principle: stable content must come before dynamic content in the prompt. The cache key is a prefix match — every token from position 0 to position N must be identical. A single changed token at position 100 invalidates caching for every token from 100 onward.

The correct ordering:

1. System prompt (role, constraints, format rules)       ← most stable
2. Tool/function schemas                                  ← changes rarely
3. Reference documents / RAG context (if static)         ← changes per doc set
4. Few-shot examples (if fixed)                           ← changes rarely
5. Conversation history (prior turns)                     ← grows per turn
6. Retrieved chunks (if dynamic)                          ← changes per query
7. Current user message                                   ← always dynamic

The cache marker goes after segment 4 (or wherever your stable content ends). Everything before it is cached; everything after it is not.

flowchart TD
    SP["System Prompt\n(role + constraints + format)\n~500–2000 tokens"]
    TS["Tool Schemas\n(function definitions)\n~500–3000 tokens"]
    FS["Fixed Few-Shot Examples\n~300–1000 tokens"]
    CM["⬅ CACHE MARKER HERE"]
    CH["Conversation History\n(grows each turn)"]
    RC["Retrieved Chunks\n(per-query RAG)"]
    UT["User Turn\n(current message)"]

    SP --> TS --> FS --> CM --> CH --> RC --> UT

    style SP fill:#15803d,color:#fff
    style TS fill:#15803d,color:#fff
    style FS fill:#15803d,color:#fff
    style CM fill:#ffaa00,color:#0a0a0f
    style CH fill:#b45309,color:#fff
    style RC fill:#b45309,color:#fff
    style UT fill:#9f1239,color:#fff

Green blocks get cached by the marker shown. The brown and red blocks aren't covered by that marker — but conversation history deserves a second look. History is appended-only: turn N's prompt starts with exactly the same tokens as turn N−1's, plus a new tail. That makes it a growing stable prefix, and both providers can cache it. OpenAI's automatic longest-prefix matching picks it up with no work on your part. On Anthropic, you place an additional cache_control marker on the last message each turn (the API allows up to 4 markers) — each call reads the previous turns from cache and writes only the new tail. The genuinely uncacheable content is what changes within a position: per-query retrieved chunks and the current user message.

What breaks the cache

Dynamic content bleeding into the prefix

This is responsible for the majority of cache-rate failures I've seen in production. The patterns:

Timestamps in the system prompt. A system prompt that starts with "Today is {current_date}" or "Current time: {timestamp}" guarantees zero cache hits. The date changes daily; the time changes every second. Move temporal context to the user turn: [System: static instructions] ... [User: (context: date=2026-07-02) What's the weather?]

User identifiers or session state. "You are assisting user John (ID: u_4829)." Every user gets their own cache entry that's only hit if the same user makes another request within the TTL. For a support bot with 10,000 unique users/day and 5-minute TTLs, the cache hit rate is approximately zero. Move personalization to the user turn, or keep the system prompt generic.

Rotating disclaimers or A/B test variants. If you're running a test where 50% of users see "Please be concise" and 50% see "Feel free to elaborate," you have two cache pools, each half as hot. This might be acceptable — model behavior genuinely differs — but it's worth knowing.

Version strings injected at build time. A CI pipeline that injects <!-- app version: 1.4.23 --> into the system prompt on every deploy resets the cache on every release. This is almost never intentional.

Insufficient prefix length

Both providers require 1,024 tokens before caching activates. If your system prompt is 400 tokens, you get no caching regardless of how well you structure it. Short system prompts are common in quick-iteration phases. For production systems with real caching ambitions, invest in system prompt depth: add explicit constraints, format rules, few-shot examples, and domain context. These also improve output quality — the caching benefit is secondary.

Low traffic endpoints

With a 5-minute TTL, an endpoint that receives one request every 7 minutes on a regular schedule never hits the cache — every gap exceeds the TTL, so every request is a fresh write. Randomly arriving traffic does better, because some gaps land inside the window; the estimator below puts numbers on it. Either way, low-traffic background jobs and batch processes rarely clear the ~22% hit rate needed to beat the write premium on Anthropic.

Cache hit rate estimator

Traffic rate:   R requests/minute
Cache TTL:      T minutes
P(hit) ≈ 1 - e^(-R×T)     (exponential distribution approximation)

At R=0.1 req/min, T=5 min: P(hit) ≈ 1 - e^(-0.5) ≈ 39%
At R=1 req/min,   T=5 min: P(hit) ≈ 1 - e^(-5)   ≈ 99.3%
At R=10 req/min,  T=5 min: P(hit) ≈ 1 - e^(-50)  ≈ ~100%

Breakeven on Anthropic (writes cost 1.25× base, hits 0.1×):
  Expected cost per request = p × 0.1 + (1 − p) × 1.25   (in units of base price)
  Caching wins when that's < 1  →  p > 0.25/1.15 ≈ 22% hit rate
  Below ~22%, the write premium costs more than the hits save
  → Plus the ROI question: restructuring effort vs savings

Agentic workloads: where caching earns the most

Single-turn chatbots are the obvious case, but multi-step agents are where caching compounds. Consider an agent that browses a codebase, reads documentation, calls tools to run tests, and produces a fix — a session with 15 LLM calls.

In each call, the agent sends:

• System prompt: 1,500 tokens (stable across the session)
• Tool schemas for 8 tools: 3,000 tokens (stable across the session)
• Conversation history + tool outputs so far: grows from 0 to ~8,000 tokens
• Current task: ~200 tokens (dynamic)

The first call is a cache miss (4,700 tokens of stable content stored). With a single marker on the system/tools block, calls 2–15 hit the cache on those 4,700 tokens; call 14 pays for 200 (current task) + 7,800 (accumulated history, uncached) + 4,700 × 10% (cached) = ~8,470 token-equivalents instead of ~12,700 — a 33% reduction. But that leaves the biggest block on the table. Because history is appended-only, you can cache it incrementally: place a second cache_control marker on the last message each turn. Now call 14 reads system + tools + all history through call 13 (~12,300 tokens) from cache at 10%, and pays the 1.25× write premium only on the ~200 new tokens: ~1,230 + ~250 = ~1,480 token-equivalents. That's roughly 88% off, versus 33% for the single-marker version. OpenAI's automatic caching gives you the incremental behavior for free — the longest matching prefix grows with the conversation.

Research on cache stability in agentic trajectories (arxiv 2601.06007) found that hit rates above 70% are achievable when the stable prefix is maintained and new content is appended rather than inserted. The failure mode is tool output injection: if a tool returns data and the orchestrator inserts it before the tool schema block rather than after, it busts the cache on every call.

# Agent loop that preserves cache-stable prefix
def run_agent_step(
    client: anthropic.Anthropic,
    conversation: list[dict],
    tools: list[dict],
    system_with_cache: list[dict],  # Pre-built with cache_control
) -> anthropic.types.Message:
    """
    Always append new messages to conversation tail.
    Never modify system_with_cache during a session.
    """
    return client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=4096,
        system=system_with_cache,   # Stable — cache hits here
        tools=tools,                # Stable — cache hits if marked
        messages=conversation,      # Grows — but stays after cache boundary
    )


def build_cached_system(system_text: str) -> list[dict]:
    return [
        {
            "type": "text",
            "text": system_text,
            "cache_control": {"type": "ephemeral"},
        }
    ]

For RAG-based agents, the question is whether retrieved documents are stable across calls. Per-query retrieval produces different chunks each call — don't cache those. But if you have a reference corpus that all calls in a session share (a user's codebase, a fixed knowledge base), load it once into a cache-marked block at session start.

What breaks (failure modes)

Silent zero hit rate

The most dangerous failure: you add cache_control, you see no errors, your costs don't drop, and you don't notice for weeks because you're not monitoring cache_read_input_tokens. Always instrument hit rate explicitly:

def log_cache_metrics(usage: anthropic.types.Usage, endpoint: str) -> None:
    total_input = (
        usage.input_tokens
        + usage.cache_creation_input_tokens
        + usage.cache_read_input_tokens
    )
    hit_rate = usage.cache_read_input_tokens / max(total_input, 1)
    
    # Emit to your observability stack
    metrics.gauge("llm.cache.hit_rate", hit_rate, tags={"endpoint": endpoint})
    metrics.increment("llm.cache.read_tokens", usage.cache_read_input_tokens)
    metrics.increment("llm.cache.creation_tokens", usage.cache_creation_input_tokens)

Alert if hit rate drops below 50% on a high-traffic endpoint. A sudden drop usually means a deploy injected something dynamic into the prefix.

Cache warming costs on cold starts

On a cold deploy, every instance's first request pays the cache write price — on Anthropic, 1.25× the base input rate (2× for the 1-hour TTL). For endpoints that handle traffic spikes with auto-scaling, new instances start cold. The write premium means caching is not free insurance: a workload that writes but rarely hits (below roughly a 22% hit rate) pays more than it would without caching. For steady production traffic this is noise, but don't be surprised when a deploy plus traffic spike shows elevated costs for a few minutes.

TTL mismatch with traffic patterns

Overnight at 2am, traffic drops to near zero and the cache expires. At 9am when traffic resumes, all requests are cache misses for the first few minutes until the cache is warm again. The cost impact is small (a few minutes of full-price requests), but latency spikes as prefill runs at full cost. If you have TTFT SLOs during ramp-up periods, explicit cache warming — a single keep-alive request to each endpoint before traffic peaks — is worth the effort.

Prompt schema changes mid-session for agents

If your agent can dynamically add tools during a session (some orchestration patterns do this), every tool schema addition changes the token sequence and busts the cache for subsequent calls. Fix: determine the full tool set at session start and keep it fixed. Load all potentially-needed tools upfront. The token overhead of an unused tool schema is a rounding error compared to the cache savings.

{"type": "token-cost", "title": "Monthly cost waterfall: cached vs uncached"}

Comparing the providers

Dimension	Anthropic	OpenAI GPT-4o	Google Gemini 2.x
Discount on cached tokens	90% off	50% off	75% off compute
Pricing model	Cached input: 10% of base rate	Cached input: 50% of base rate	Storage charge + reduced compute
Min prefix to cache	1,024 tokens	1,024 tokens	Order of a few thousand tokens (model-dependent, mid-2026)
Configuration	Explicit cache_control markers	Automatic	Both: implicit (automatic) and explicit caching API
Cache TTL	5 min, refreshes on hit; 1-hr tier at 2× write	5–10 min inactivity	Up to hours with explicit TTL
Multiple cache points	Yes (up to 4 markers)	No (longest prefix only)	Yes
Latency benefit	~85% TTFT reduction on hits	Moderate	Moderate
Hit rate visibility	cache_read_input_tokens	cached_tokens in usage	Usage metadata

If you have the option to use Anthropic's implementation, the 90% discount versus OpenAI's 50% is meaningful at scale — roughly twice the savings on the cached portion. The explicit markers also give you debugging leverage: you can check whether cache creation tokens appear in the usage, which tells you whether your marker was recognized. With OpenAI's automatic caching, a miss is silent.

Google's Gemini offers two mechanisms: implicit caching that works automatically like OpenAI's, and an explicit context caching API where you upload content to a cache object with an explicit TTL (up to hours) and reference it in subsequent requests. Minimums are on the order of a few thousand tokens as of mid-2026, model-dependent — check the current docs before designing around a threshold. The explicit API shines for very large shared corpora (entire codebases, long documents) reused across many sessions, where you control the cache lifetime instead of racing an inactivity window.

For systems already invested in a provider's ecosystem, the practical advice is: use whichever caching mechanism that provider offers and focus engineering time on the structural discipline. A well-structured prompt gets 70–80% cache hit rates on any provider; a poorly structured one gets near zero regardless of the discount rate.

Worked example: refactoring a broken prompt

Here's a real failure pattern and its fix. Original prompt (schematic — abbreviated):

BROKEN: cache hit rate ≈ 0%

system: |
  You are a customer support agent for Acme Corp.
  User: {user.first_name} (ID: {user.id})
  Session started: {datetime.now()}
  Your role is to help with billing, shipping, and returns.
  [... 800 more tokens of policy and constraints ...]
  Always respond in {user.preferred_language}.

Every field in {} is dynamic. The system prompt changes on every request. Cache hit rate: zero.

FIXED: cache hit rate ≈ 85%

system (with cache_control): |
  You are a customer support agent for Acme Corp.
  Your role is to help with billing, shipping, and returns.
  [... 800 tokens of static policy and constraints ...]
  Always respond in the language the user writes in.
  # CACHE MARKER HERE

user turn: |
  [Context: user={user.first_name}, session_id={session_id}, 
   started={datetime.now()}, language={user.preferred_language}]
  
  User message: {actual_user_message}

The static policy block — 850 tokens — now gets cached. The user-specific data moves to the user turn where it belongs. The instruction "respond in the language the user writes in" is often equivalent to "respond in {user.preferred_language}" and doesn't require dynamic injection.

For system prompt design patterns that make this restructuring easier from the start, see the companion article — building cache-friendly prompts is much less painful when the system prompt is modular from the beginning.

The judgment call: when to invest in caching

Caching is always net-positive if your prefix already exceeds 1,024 tokens and your traffic exceeds one request per minute. The structural discipline has zero downside once you understand the rule. The question is priority.

Invest now if:

• Your system prompt or tool schemas exceed 2,000 tokens (the cache savings compound heavily)
• You're running a multi-step agent with 10+ turns per session
• Your endpoint sees more than 10 requests/minute
• You're on Anthropic's API where the 90% discount makes even moderate traffic worthwhile

Defer if:

• Your system prompt is under 500 tokens and you're not ready to expand it
• Traffic averages less than one request per 10 minutes (the estimator above puts the hit rate near the ~22% breakeven, so there's little to gain)
• You're in active iteration on the system prompt (restructuring constantly resets the benefit)
• You're running a batch job that processes unique documents per request (dynamic RAG content won't cache)

One thing worth tracking: cost optimization in LLM systems is often better attacked through model routing — sending cheaper models the queries that don't need frontier capability — and through context window management to avoid sending 10,000 tokens when 2,000 would do. Prompt caching is orthogonal to both and stacks with them. Get your context architecture clean first (see context engineering foundations), then apply caching as the final cost discipline.

The teams that do this well treat it as part of their token economics discipline — monitoring cache hit rates alongside token counts, flagging drops, and reviewing prompts for cache-busting patterns the same way they review code for performance regressions. The teams that don't do it are paying 2–5× what they need to for input tokens, and the bill grows with every user they acquire.

That's the most frustrating part: the savings compound with success. The more popular your product, the more you pay for a structurally broken prompt. Fix it once, and every new user is 70–80% cheaper to serve.