Prompt caching deep dive: prefix caching, semantic caching, and when each wins

The bill was fine until it wasn't. Six months after launching an internal knowledge-base chatbot, the team noticed their API spend tracking 4x higher than expected at the same query volume. The culprit turned out to be three things layered on top of each other: a 2,000-token system prompt being re-sent fresh on every call, a set of 20 tool schemas injected at the top of each request, and a session history that grew to 15 turns before being summarized. Nothing broke. The quality was fine. But every single token in that 6,000-token context was being priced as fresh input, on every request, all day long.

Prompt caching is the structural fix, and it comes in two distinct forms that most engineers conflate until they have a bill like this to explain.

What prefix caching actually is

When an LLM processes a prompt, the transformer computes key-value attention activations for every input token. These KV vectors are what the model "remembers" as it generates output. For any fixed prefix — a system prompt, a set of tool schemas, a block of retrieved documents — these activations are identical every time you send the same prefix. Recomputing them from scratch on every call wastes compute proportional to the prefix length.

Prefix caching solves this by storing the KV activations for a fixed leading prefix and reusing them on subsequent requests. The cache key is an exact prefix match: every token in the prefix must match exactly, in order, up to the cache breakpoint. The model then only runs the full prefill computation for the tokens that follow the cached prefix.

sequenceDiagram
    participant App as Application
    participant API as Provider API
    participant KV as KV Cache (provider-side)

    App->>API: Request 1: [system(1200t) + context(3000t) + query_1(50t)]
    API->>KV: Compute and store KV for first 4200 tokens
    API-->>App: Response 1 (full prefill cost)

    App->>API: Request 2: [system(1200t) + context(3000t) + query_2(50t)]
    API->>KV: Prefix match — retrieve cached KV for 4200 tokens
    API-->>App: Response 2 (cache read rate on 4200t, fresh rate on 50t only)
    Note over App,KV: 90% cost reduction on the cached segment

Anthropic requires you to mark cache breakpoints explicitly using a cache_control field:

import anthropic

client = anthropic.Anthropic()

system_with_cache = [
    {
        "type": "text",
        "text": SYSTEM_PROMPT,  # your stable system prompt
        "cache_control": {"type": "ephemeral"},
    },
    {
        "type": "text",
        "text": RAG_CONTEXT,    # retrieved documents or static context
        "cache_control": {"type": "ephemeral"},
    }
]

response = client.messages.create(
    model="claude-sonnet-4-5-20251001",
    max_tokens=1024,
    system=system_with_cache,
    messages=[{"role": "user", "content": user_query}]
)

# Check what was cached
usage = response.usage
print(f"Cache read tokens: {usage.cache_read_input_tokens}")
print(f"Cache write tokens: {usage.cache_creation_input_tokens}")
print(f"Fresh input tokens: {usage.input_tokens}")

The first request populates the cache (cache_creation_input_tokens) — and writes are not free: Anthropic charges a 25% premium on cache-write tokens, ~$3.75/M at the illustrative $3.00/M base rate. Every subsequent request with the same prefix gets the cache read rate. The default 5-minute TTL refreshes every time the cached prefix is used — reads included — so any route with steady traffic keeps its cache warm indefinitely. If your calls are further apart than that, Anthropic also sells a 1-hour TTL at a 2× write multiplier. For chatbots and agents with sustained conversation, the default works well; for batch workloads where jobs are hours apart, either pay for the longer TTL or accept cold writes between runs.

OpenAI handles this automatically: any request where the prompt shares a prefix of at least 1,024 tokens with a recent request gets a 50% discount on those tokens, with no code changes. Google Gemini's explicit caching API (cachedContent) also requires a 2,048-token minimum and charges a small storage fee per token-hour for cached content — worth it for very long documents referenced repeatedly.

The math on when prefix caching pays

The savings are real, but only under the right conditions:

Break-even analysis for prefix caching

Variable:
  prefix_tokens = P      (the stable content you are caching)
  fresh_rate    = $3.00/M (illustrative Anthropic Sonnet input)
  cache_rate    = $0.30/M (illustrative Anthropic cache read)
  write_rate    = $3.75/M (cache write, 25% premium over fresh)
  hit_rate      = H       (fraction of requests that hit the cache)

Cost with no caching per request (prefix P + query Q tokens):
  (P + Q) × $3.00/M

Cost with caching at hit rate H:
  (1 - H) × (Q × $3.00/M + P × $3.75/M)   [misses pay the write premium on P]
  + H × (Q × $3.00/M + P × $0.30/M)       [hits read the cached prefix]

Break-even: each hit saves $2.70/M on P; each miss costs an extra $0.75/M on P.
  Net positive when H × 2.70 > (1 - H) × 0.75  →  H > ~0.22.
  Below a ~20% hit rate, caching loses money at these rates.
Above that, the question is magnitude:

P = 4,200 tokens, Q = 50 tokens, H = 0.90:
  No cache: 4,250 × $3.00/M = $0.01275/req
  Cached:   0.10 × (50 × $3.00 + 4,200 × $3.75) + 0.90 × (50 × $3.00 + 4,200 × $0.30)
          = 0.10 × $0.0159 + 0.90 × $0.00141 = $0.00159 + $0.00127 = $0.00286/req
  Savings: ~78% per request

P = 500 tokens, Q = 50 tokens, H = 0.90:
  No cache: 550 × $3.00/M = $0.00165/req
  Cached:   0.10 × (50 × $3.00 + 500 × $3.75) + 0.90 × (50 × $3.00 + 500 × $0.30)
          = 0.10 × $0.002025 + 0.90 × $0.00030 = $0.0002 + $0.00027 = $0.00047/req
  Savings: ~71% — $0.00047/req vs $0.00165/req
  At 100k calls/day: ~$47/day vs $165/day — roughly $3,500/month in absolute savings.
  Worth doing even for small prefixes.

The practical lesson: prefix caching is worth implementing any time your fixed context exceeds 1,024 tokens and the same prompt structure appears across multiple requests. That covers almost every production use case with a non-trivial system prompt.

{ "type": "prompt-cache", "scenario": "chatbot", "title": "Prefix cache anatomy: stable vs variable segments" }

Structuring prompts for maximum cache hits

This is where most teams leave money on the table. The cache key is an exact prefix — any variable content at the start of the prompt resets it. The structural rule is simple: stable content first, dynamic content last.

WRONG — cache-busting structure:
  1. "Today is July 2, 2026. Session ID: abc-123."  ← variable, busts cache immediately
  2. [2,000-token system prompt]
  3. [3,000-token RAG context]
  4. User message

RIGHT — cache-friendly structure:
  1. [2,000-token system prompt]                     ← always identical, cached
  2. [3,000-token RAG context]                       ← stable for this query type, cached
  3. User message                                    ← variable, never cached

Injecting a timestamp, request ID, or randomized nonce at the top of the system prompt is the single most common prefix caching anti-pattern. The session ID belongs in the user message turn, not the system prompt. If your RAG pipeline injects retrieved documents at different positions depending on query type, you are also fragmenting your caches — retrieved context should inject at a consistent position after the system prompt.

Tool schemas are often overlooked. If you use function calling, the tool definitions sent with each request add hundreds to thousands of tokens. Put them immediately after the system prompt, before the conversation history, and they get cached on the first call. This is especially valuable for agents that carry 20+ tool schemas per request.

What semantic caching actually is

Semantic caching operates one layer above the LLM. Instead of making the LLM call cheaper, it skips the LLM call entirely by recognizing that a new query is similar enough to a previous one that the stored response is still valid.

The mechanics: embed the incoming query using an embedding model, search a vector store for the nearest stored query-response pair, and if the cosine similarity exceeds a threshold, return the stored response. If not, call the LLM, store the response with its embedding, and return it.

from openai import OpenAI
import numpy as np

client = OpenAI()

# Simplified semantic cache (use Redis or Qdrant in production)
cache: list[dict] = []

def cosine_similarity(a: list[float], b: list[float]) -> float:
    a_arr = np.array(a)
    b_arr = np.array(b)
    return float(np.dot(a_arr, b_arr) / (np.linalg.norm(a_arr) * np.linalg.norm(b_arr)))

def embed(text: str) -> list[float]:
    resp = client.embeddings.create(model="text-embedding-3-small", input=text)
    return resp.data[0].embedding

def query_with_semantic_cache(user_query: str, threshold: float = 0.92) -> str:
    query_embedding = embed(user_query)

    # Check cache
    for entry in cache:
        similarity = cosine_similarity(query_embedding, entry["embedding"])
        if similarity >= threshold:
            print(f"Cache hit (similarity={similarity:.3f})")
            return entry["response"]

    # Cache miss — call the model
    print("Cache miss — calling LLM")
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": user_query}]
    )
    text = response.choices[0].message.content

    # Store in cache
    cache.append({
        "query": user_query,
        "embedding": query_embedding,
        "response": text,
    })
    return text

Production implementations use a proper vector store (Qdrant, Redis with the vector module, pgvector) and add TTL metadata, versioning, and scope tags to each entry. The embedding call itself costs roughly $0.02/M tokens with text-embedding-3-small — essentially free compared to LLM input tokens, which makes the overhead negligible for any hit rate above a few percent.

When hit rates are actually high

The published benchmark of 60–65% latency reduction at 67% hit rate comes from controlled domains. Real-world hit rates vary sharply:

Query type	Typical hit rate	Why
FAQ / help center lookups	55–75%	Users ask the same questions in slightly different phrasings
Code snippet generation (narrow domain)	45–65%	Embedding space is dense for constrained vocabularies
Internal knowledge base Q&A	30–55%	Depends on how canonical the question set is
Customer support (product-specific)	25–50%	High query repetition by issue type
Open-ended chat / creative tasks	5–15%	Embeddings spread sparsely; almost nothing matches
Personalized recommendations	Near 0%	Correct answer depends on user state not in the embedding

The implication is that semantic caching is worth measuring, not worth assuming. Instrument your cache with hit rate per route from day one. If a route has a 5% hit rate, the cache lookup overhead (one embedding API call per request) is costing you more than it saves.

{ "type": "token-cost", "title": "Cost waterfall: uncached vs prefix-cached vs semantic cache hit" }

What breaks

Prefix caching failure modes

Cache expiry between related calls. Anthropic's default 5-minute TTL is adequate for real-time chat but inadequate for scheduled batch jobs that process items an hour apart. Those jobs pay full price on every request regardless of how identical their system prompts are. The workaround is to send a warm-up call before a batch run starts, then ensure the batch completes within the TTL window — or restructure offline workloads to use the Batch API endpoint, which has its own pricing advantages.

Prompt engineering that fights the cache. Teams that use prompt management systems often inject metadata (run IDs, AB test variants, user cohort tags) into the system prompt. Every unique injection is a unique prefix that never caches. Move runtime metadata to the user turn or to HTTP headers — keep the system prompt byte-for-byte identical across all calls in the same logical variant.

RAG context that changes per query. If your retrieval layer injects different documents for different queries into a position before the user message but after the system prompt, you get partial cache hits: the system prompt caches but the variable document block does not. This is often fine and still valuable — the system prompt alone being 1,200 tokens delivers meaningful savings. But if you want to cache the documents too, you need to retrieve a fixed superset of documents per query class rather than a variable per-query set.

Semantic caching failure modes

Stale entries serving wrong answers. This is the category that silently costs you user trust. A cached response about your return policy was correct when it was stored. Your policy changed. The cache does not know. The next user who asks a similar-enough question gets the old answer with full confidence. You have no error signal.

The fix is explicit cache management: every entry needs a TTL appropriate to its content type and a version key tied to the underlying data source. When your knowledge base changes, increment the version, and old entries become invalid on the version check before you ever return them. For truly dynamic content (live pricing, current stock, news), do not semantic-cache at all.

Threshold tuning as a false sense of precision. A threshold of 0.95 feels conservative. But "conservative" is relative to your embedding model, your content domain, and whether you have adversarial inputs. 2025 research demonstrated that adversarial prompts crafted to land within the cosine-similarity threshold of a stored query can retrieve responses for different queries — including responses that contain sensitive content from prior sessions. Your threshold must be tuned with adversarial test cases, not just semantic accuracy test cases. This matters most if your cache can contain user-private data.

flowchart TD
    ENTRY[Cache entry created] --> TTL{TTL expired?}
    TTL -->|Yes| MISS[Cache miss\ncall LLM]
    TTL -->|No| VER{Version\ncurrent?}
    VER -->|Stale| MISS
    VER -->|Current| SIM{Similarity\n≥ threshold?}
    SIM -->|No| MISS
    SIM -->|Yes| ADV{Adversarial\ncheck passes?}
    ADV -->|No| MISS
    ADV -->|Yes| HIT[Cache hit\nreturn stored response]
    style HIT fill:#15803d,color:#fff
    style MISS fill:#ff2e88,color:#fff

Per-user personalization defeats domain-level caching. If the correct answer to "What is my balance?" or "What is my subscription tier?" depends on which user is asking, any similarity-based cache that does not include the user identity in the cache key will return wrong data for different users. Add user scope to cache lookups or exclude personalized routes explicitly.

Cache stampede on cold start. If your semantic cache is empty and a burst of 1,000 similar queries arrives simultaneously, all of them miss, all of them call the LLM, and all of them store identical or near-identical responses. You have wasted 999 LLM calls. The fix is a request-coalescing layer: for in-flight queries, deduplicate on the query hash and have the second-through-Nth requests wait for the first to complete and populate the cache.

Combining the two layers

The two caching strategies are not in competition — they address different points in the cost structure.

flowchart LR
    subgraph "Layer 1: Application (semantic cache)"
        Q[Query] --> EMB[Embed query]
        EMB --> VEC[(Vector store\nquery-response pairs)]
        VEC -->|Hit| RET[Return cached response\n~1ms + embedding cost]
        VEC -->|Miss| LYR2
    end
    subgraph "Layer 2: Provider (prefix cache)"
        LYR2[LLM call with\ncache-structured prompt] --> PROV[Provider API]
        PROV -->|Prefix cached| CHEAP["Cached rate\n($0.30/M on prefix)"]
        PROV -->|First call| WRITE["Write cache\n($3.00/M, populates cache)"]
        CHEAP --> RESP[Response]
        WRITE --> RESP
    end
    RESP --> STORE[Store in semantic cache]
    RESP --> OUT[Return to user]

The practical combination for a customer support application might be:

1. Check semantic cache first. If the query is within threshold of a stored answer, return it immediately. No LLM cost, ~1ms latency plus the embedding call.
2. On a cache miss, call the LLM. Because the system prompt and static tool schemas are structured as a stable prefix, the provider caches their KV activations and charges the cache read rate.
3. Store the LLM response in the semantic cache with a TTL appropriate to the content — 24 hours for stable FAQ answers, 1 hour for pricing-adjacent content, never for personalized responses.

The combined savings on a route with 40% semantic cache hits and 90% prefix cache hits among the misses are substantial. If fresh cost is $0.0158/call (from the earlier estimate):

Combined caching math (same chatbot example):
  40% of calls: semantic cache hit → ~$0.0001/call (embedding only)
  60% of calls go to LLM:
    Of these, 90% hit prefix cache → $0.0044/call
    Of these, 10% are cold (first call or expired) → $0.0158/call

Blended cost:
  0.40 × $0.0001 + 0.60 × (0.90 × $0.0044 + 0.10 × $0.0158)
= $0.00004 + 0.60 × (0.00396 + 0.00158)
= $0.00004 + 0.60 × $0.00554
= $0.00004 + $0.00332
= $0.00336/call

vs $0.0158/call uncached → 79% reduction
At 500k calls/month: $7,900 → $1,680

Measuring what matters

The single most common instrumentation mistake is measuring a single global cache hit rate. A product with three routes — a high-traffic FAQ lookup (60% hit rate), a medium-traffic code assistant (30% hit rate), and a low-traffic creative writing feature (3% hit rate) — might show a 35% blended hit rate that looks reasonable. But the creative writing route may be burning 40% of total spend with effectively zero caching benefit and deserves either different tooling (no semantic cache, optimized prefix structure) or a different model tier entirely. See token economics for how to break down spend by route before optimizing.

Minimum metrics set for a production caching layer:

• Semantic cache hit rate per route, sampled at 1-minute granularity
• Average cosine similarity of cache hits per route (a sustained drop here means your query distribution shifted)
• Prefix cache token fraction: cache_read_tokens / (cache_read_tokens + input_tokens) per request type
• p50/p95/p99 latency on cache hits vs misses (to verify the latency benefit is real in your stack)
• Periodic correctness audits on cache hits: sample 1% of served responses and check for staleness

The last one is the hardest and the most important. Without it, stale entries accumulate invisibly until a user reports something wrong. Plug this into your broader observability stack alongside model response quality signals.

The decision in practice

Use prefix caching universally — it requires only prompt structure discipline and is almost always net positive above 1,024 stable tokens. There is no meaningful downside other than the engineering cost of restructuring prompts that were originally written without caching in mind.

Use semantic caching selectively. Start with routes that have stable, factual content and high query repetition (FAQ, internal knowledge base, code pattern lookup). Exclude personalized responses, time-sensitive content, and any query type where you cannot define a sensible TTL. Do not deploy semantic caching for creative or open-ended tasks where the hit rate will be negligible.

When both are in place, the LLM gateway layer is a natural home for both: it sees every request, can embed queries and check a semantic cache before routing to the provider, and can enforce per-route cache policies with consistent observability. LiteLLM implements prefix caching passthrough and has pluggable semantic cache backends; Portkey's open-sourced gateway (Apache 2.0 since March 2026) ships semantic caching with configurable similarity thresholds out of the box.

The cases where neither caching approach helps are worth naming: highly personalized single-user queries, real-time data lookups, and workloads where every request is genuinely unique (creative generation, document summarization for novel documents). For those, the right optimization levers are model routing to cheaper models and streaming and batching to shape throughput. Caching is not a universal solution — it is the first thing to reach for when your query distribution has any repetition at all.

The teams that get the most out of caching treat prompt structure as load-bearing architecture. Once you internalize that every token in a fixed prefix is eligible for caching and every variable token at the top is an anti-pattern, you start writing prompts differently — and the bill reflects it.