Chunking Strategies That Actually Matter: From Fixed-Size to Late Chunking

A team I know built a legal document search system and shipped it with 1,024-token fixed-size chunks because their initial demo looked good. Four weeks later, support tickets confirmed the system was confidently surfacing contract clauses stripped of the definitions section they depended on. The clause "shall not exceed the Permitted Amount" appeared in retrieval with high confidence. The definition of "Permitted Amount" was 1,100 tokens earlier in the same document — in a different chunk entirely — and never retrieved. The embedding model had no idea what it was missing. Rewriting the retrieval query, adding reranking, switching embedding models: none of it helped, because the problem was upstream of all of that.

Chunking is the decision that determines what information your embedding model ever gets to see. Get it wrong and the rest of the pipeline is optimizing noise.

What chunking actually does to your embeddings

An embedding model maps a text sequence to a fixed-size vector. That vector is a compressed representation of everything the model understood about the sequence. The compression is lossy. A 512-dimensional vector cannot capture every fact in a 1,000-token passage with equal fidelity — the model learns to represent the "gist," weighted toward tokens and phrases that appear often in training.

When you chunk a document before embedding, each chunk is embedded in isolation. The embedding model has no knowledge of what appeared before or after. A pronoun ("it") loses its antecedent. A number ("$42 million") loses its label ("quarterly revenue"). A clause loses the definition it refers to. These are not edge cases — they are structural features of any document with more than a few paragraphs.

This matters for retrieval because the embedding distance between a query and a chunk depends entirely on what both vectors represent. If the chunk vector is missing the entity that the query is asking about (because that entity was defined in an adjacent chunk), the distance will be high regardless of semantic relevance.

sequenceDiagram
    participant Q as User query
    participant E as Embedding model
    participant V as Vector store
    participant C as Chunk A (isolated)
    participant D as Chunk B (has definition)

    Q->>E: embed("what is the permitted amount?")
    E-->>Q: query vector

    C->>E: embed("shall not exceed the Permitted Amount")
    note over C,E: No context about what "Permitted Amount" means
    E-->>C: chunk vector (missing definition context)

    D->>E: embed("Permitted Amount means $2.5M per quarter")
    E-->>D: chunk vector (has definition)

    Q->>V: ANN search
    V-->>Q: returns Chunk A (high cosine sim on surface terms)
    note over Q,V: Wrong chunk retrieved — definition never found

The fix is not always better embeddings or a better model. Often it is better chunking.

Fixed-size chunking: the baseline you need to understand

Fixed-size chunking splits a document into segments of N tokens with an optional overlap. It is the baseline because it is fast, deterministic, and requires no ML beyond the embedding step itself.

from langchain.text_splitter import RecursiveCharacterTextSplitter

# from_tiktoken_encoder makes chunk_size count TOKENS. The plain
# constructor counts characters — 512 chars is only ~128 tokens.
splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""]
)

chunks = splitter.split_text(document_text)

The RecursiveCharacterTextSplitter from LangChain tries each separator in order, falling back to the next if a split would exceed the chunk size. One footgun worth naming: the default constructor measures chunk_size in characters, not tokens, so RecursiveCharacterTextSplitter(chunk_size=512) gives you ~128-token chunks — a quarter of what every sizing recommendation in this article means. Use from_tiktoken_encoder (or pass a token-counting length_function) so the number you configure is the number you reason about. This is a meaningful improvement over a naive character count: you're much less likely to split mid-sentence. But you're still splitting at arbitrary positions relative to semantic structure, and the overlap window (64-128 tokens is typical) is a crude approximation of context preservation.

What breaks: a 512-token chunk that starts mid-paragraph carries no heading context. The embedding model has no way to know what section of the document this is from. A query asking about "pricing in section 4.2" retrieves a chunk that could be from section 3 or section 5 with nearly equal probability.

The 256-512 token range is where most production RAG systems land. Below 100 tokens, a chunk rarely contains enough substance for the LLM to generate a useful answer — you're essentially retrieving sentence fragments. Above 1,000 tokens, relevance scores degrade because the embedding is averaging across too many topics.

{ "type": "rag-pipeline", "title": "Fixed-size chunking in the RAG pipeline", "query": "policy" }

Recursive character splitting: the practical upgrade

The real LangChain default isn't pure fixed-size — it is recursive character splitting with ["\n\n", "\n", ". ", " ", ""] as separators. Paragraph breaks and sentence ends take priority; character splits are a last resort. This alone avoids most mid-sentence splits.

For most teams building a first production RAG system, this is where to start and where to stay until you have retrieval metrics that tell you otherwise. It is fast, it handles most document formats, and the failure modes (cross-boundary references, missing heading context) are predictable enough to work around with the other techniques covered below.

Structure-aware splitting is the next natural step for structured documents. Markdown files have headers. PDFs have sections. HTML has <h2> and <p> tags. Using the document's own structure as chunk boundaries preserves section context — "this chunk is from the 'Cancellation Policy' section" — without any ML at all. LlamaIndex's MarkdownNodeParser and the HTML splitters in LangChain are practical implementations.

Semantic chunking: boundaries from cosine similarity

Semantic chunking abandons fixed sizes and finds natural topic boundaries in the text. The standard approach:

1. Split the document into sentences.
2. Embed each sentence (or a sliding window of 3 sentences) independently.
3. Compute cosine similarity between consecutive sentence embeddings.
4. Where similarity drops below a threshold, insert a chunk boundary.

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

chunker = SemanticChunker(
    OpenAIEmbeddings(model="text-embedding-3-small"),
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=95,  # split at the 95th-percentile similarity drop
)

chunks = chunker.split_text(document_text)

The quality improvement for narrative text is real. Chapters, argument sections, and topic shifts that paragraph breaks miss get caught by the similarity drop. The resulting chunks tend to cover a coherent idea rather than an arbitrary token window.

The costs are equally real. Embedding every sentence before chunking is 10-30x slower than recursive character splitting. For a 1,000-page corpus, that ingestion slowdown is measured in hours, not minutes. And the similarity threshold requires tuning per corpus — what counts as a "topic shift" in a dense academic paper is different from what counts as one in a customer support transcript.

Semantic chunking also fails on corpora where topic transitions are gradual. Policy documents that slide from one clause to the next without a sharp boundary, code files that interleave function definitions with class context, and database schema files all produce noisy similarity signals. The tool makes a false promise of "natural" boundaries when the document doesn't have them.

Late chunking: embed the document, then apply the boundaries

Late chunking (introduced in JinaAI's 2024 late-chunking paper on their long-context v2 models, then shipped as an API flag with jina-embeddings-v3) inverts the standard order of operations. Instead of chunking first and embedding the chunks, you embed the full document first — using a long-context embedding model — and then apply chunk boundaries to the resulting token embeddings via mean-pooling.

flowchart TD
    subgraph Standard chunking
        A1[Document] --> B1[Split into chunks]
        B1 --> C1[Embed each chunk independently]
        C1 --> D1[Each chunk vector = isolated context]
    end

    subgraph Late chunking
        A2[Document] --> B2[Embed full document\n long-context model]
        B2 --> C2[Apply chunk boundaries\n via mean-pooling]
        C2 --> D2[Each chunk vector = document-aware context]
    end

    style D2 fill:#a855f7,color:#fff
    style D1 fill:#ff2e88,color:#fff

The key property: token embeddings produced by a transformer are contextual — each token's representation is influenced by every other token the model attended to. When you mean-pool a span of token embeddings from a full-document pass, the resulting vector carries information from the entire document, not just the local 512-token window.

In practice, the "Permitted Amount" problem from the opening disappears. The clause "shall not exceed the Permitted Amount" gets embedded with a vector that already absorbed the definition 1,100 tokens earlier, because the full-document pass saw both. The resulting chunk vector is distinguishable from unrelated clauses in a way that the isolated-embedding version is not.

In Jina's published evaluation, late chunking outperforms naive chunking on most BEIR subsets, with the biggest gains on long documents and the smallest on short-document sets. It has not been benchmarked head-to-head against semantic chunking or Contextual Retrieval, so treat "late beats flat" as a claim about naive splitting, not about every alternative. The main constraint is the context window of the embedding model. jina-embeddings-v3 supports 8,192 tokens, which handles most documents but not book-length corpora. For documents exceeding the model's context window, late chunking degrades to standard chunking at the truncation boundary.

import requests

def late_chunk(chunks: list[str]) -> list[list[float]]:
    """
    Late chunking via Jina's API. You split the document into chunk
    strings yourself; each element of `input` is one chunk of a
    single document, in order.
    """
    response = requests.post(
        "https://api.jina.ai/v1/embeddings",
        json={
            "input": chunks,
            "model": "jina-embeddings-v3",
            "task": "retrieval.passage",
            # Jina concatenates the inputs, embeds the full sequence,
            # then mean-pools per supplied chunk — server-side pooling,
            # not boundary detection. Boundaries are yours to choose.
            "late_chunking": True,
        },
        headers={"Authorization": f"Bearer {JINA_API_KEY}"},
    )
    # One embedding per input chunk, each carrying full-document context
    return [item["embedding"] for item in response.json()["data"]]

JinaAI's API exposes late chunking directly; open-source models require pulling token embeddings and pooling manually. The Graph-Aware Late Chunking extension (ArXiv 2026) adds structural and relational metadata to late-chunked embeddings, combining document context with relational graph signals — though that is currently a research artifact rather than production tooling.

Contextual Retrieval: the LLM-generated prefix

Contextual Retrieval (Anthropic, September 2024) takes a different approach to the same problem: instead of changing how embeddings work, it changes what text gets embedded. Before indexing each chunk, an LLM generates a 50-100 token summary that situates the chunk in its document context. This prefix is prepended to the chunk, and the combined text is embedded and BM25-indexed.

The prompt Anthropic published:

<document>
{{WHOLE_DOCUMENT}}
</document>

Here is the chunk we want to situate within the whole document:
<chunk>
{{CHUNK_CONTENT}}
</chunk>

Please give a short succinct context to situate this chunk within the overall document
for the purposes of improving search retrieval of the chunk.
Answer only with the succinct context and nothing else.

The output for our legal document example might be:

This chunk appears in Section 12.3 (Financial Limits) of the Master Services
Agreement and references the "Permitted Amount" defined in Section 2.1 as $2.5M
per quarter. It sets the cap on liability for service interruptions.

Now when a query arrives for "what is the cap on liability?", the BM25 index contains the phrase "cap on liability" and the chunk contains "Permitted Amount" with a brief definition — and both the sparse and dense retrieval signals are dramatically stronger.

The measured impact: 49% reduction in retrieval failures standalone. 67% when combined with a cross-encoder reranker. These are large numbers, and Anthropic's published evaluation is reasonably rigorous. The technique works because it fixes the fundamental information asymmetry: at query time, the retriever is trying to match a short natural-language question against chunks that use the document's own vocabulary, which may be entirely different.

The cost question is the real decision point:

Cost model: 1 million document tokens
Chunks at 512 tokens avg → ~2,000 chunks

Without prompt caching:
  Claude 3.5 Haiku at ~$0.80/million input tokens (illustrative, mid-2026)
  Full document must be in context for EVERY chunk → N × doc_length tokens
  For a 10,000-token doc with 20 chunks: 20 × 10,000 = 200,000 input tokens
  = $0.16 per document (no caching)
  At 1,000 documents: $160 just for context generation

With Anthropic prompt caching (document is the stable prefix):
  Cache write: 1× at full price
  Cache reads: (N-1)× at ~90% discount
  Effective cost: ~$1.02 per million document tokens (Anthropic's published figure)
  At 1,000 documents × 10k tokens each: ~$10.20 total

Without prompt caching, Contextual Retrieval is economically impractical for anything but small corpora. With caching, it becomes one of the highest-ROI improvements available in the RAG stack.

Parent-child chunking: decouple retrieval from generation

Parent-child chunking (also called hierarchical chunking) solves a real tension: small chunks retrieve precisely because they contain a tight semantic signal, but the LLM needs enough surrounding context to reason from what it retrieves.

The architecture is simple: one vector index of small child chunks, plus a docstore of parent chunks linked by ID.

from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Qdrant

# from_tiktoken_encoder so these sizes are tokens, not characters
parent_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=1024)
child_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(chunk_size=128)

store = InMemoryStore()  # replace with Redis or similar in production

retriever = ParentDocumentRetriever(
    vectorstore=Qdrant.from_documents(...),
    docstore=store,
    child_splitter=child_splitter,
    parent_splitter=parent_splitter,
)
# ANN search on 128-token child chunks
# Returns corresponding 1,024-token parent to the LLM

The ANN search runs against 128-token child chunks, which are precise enough to identify the right document fragment. The retrieval call then looks up the parent chunk ID and returns the surrounding 1,024-token window to the LLM. The LLM gets the reasoning context it needs. The vector store maintains retrieval precision.

The tradeoff: only the child chunks get embedded and indexed — there is no second vector index and no doubled embedding bill — but 128-token children mean roughly 4x more vectors than a flat 512-token split of the same corpus, and the docstore holding parent chunks adds storage and a lookup hop. For high-precision requirements — legal, medical, technical documentation — this tradeoff is usually worth it.

Choosing between strategies

flowchart TD
    START([New RAG project]) --> Q1{Document structure?}
    Q1 -->|Well-structured\nMarkdown / HTML| STRUCT[Structure-aware splitting\nrespect heading hierarchy]
    Q1 -->|Unstructured text\narticles / books| Q2{Ingestion speed constraint?}
    Q1 -->|Mixed / unknown| FIXED[Recursive char split\n256-512 tok baseline]

    Q2 -->|Speed matters\nbatch ingestion SLA| FIXED
    Q2 -->|Quality over speed| Q3{Long-context embed\nmodel available?}

    Q3 -->|Yes jina-v3 or similar| LATE[Late chunking\nfull-doc embed → pool]
    Q3 -->|No short-context only| SEM[Semantic chunking\nsimilarity-boundary detection]

    STRUCT --> CTX{Budget for\nContextual Retrieval?}
    FIXED --> CTX
    LATE --> CTX
    SEM --> CTX

    CTX -->|Yes + prompt caching| ENRICH[Add LLM context prefix\nper chunk before indexing]
    CTX -->|No| PARENT{Dense docs?\nPrecision critical?}

    ENRICH --> DONE([Ship with hybrid\nsearch + reranker])
    PARENT -->|Yes| HIER[Parent-child chunking\nsmall retrieve / large return]
    PARENT -->|No| DONE

    HIER --> DONE

    style LATE fill:#a855f7,color:#fff
    style ENRICH fill:#00e5ff,color:#0a0a0f
    style DONE fill:#15803d,color:#fff

Strategy	Ingestion speed	Retrieval quality	Context awareness	Best for
Fixed-size (recursive char)	Fast	Baseline	None	First cut, speed-constrained
Semantic chunking	10-30x slower	+10-20% on narrative	Local cosine signal	Articles, books, narrative text
Late chunking	Moderate (long embed)	Strong on long-doc BEIR subsets	Full document	Any corpus within embed context limit
Contextual Retrieval	Slow (LLM per chunk)	+49% failure reduction	Document-summarized	High-value corpora with prompt caching
Parent-child	More child vectors + docstore	Precision + coherence	Local parent	Dense reference docs, legal, medical

These strategies compose. Late chunking + Contextual Retrieval is a valid (and expensive) combination. Parent-child + Contextual Retrieval is the combination that works well for legal corpora where the parent chunk needs its full context prefix for both the child and parent levels.

What breaks

The boundary artifact problem. Every chunking strategy has a failure mode at chunk boundaries. Fixed-size: splits mid-sentence. Semantic: false boundary from a stylistic sentence that happens to shift cosine similarity. Late chunking: the long-context model's attention attenuates at very long distances — a 200-page document will have weaker cross-chunk context near pages 1 and 200 than in the middle. There is no chunking strategy that eliminates boundary artifacts; the goal is to minimize them and make them predictable.

Contextual Retrieval without caching. The economics only work with Anthropic's prompt caching. Running Contextual Retrieval without caching on a corpus that changes daily — where the document-stable prefix cannot be cached — costs an order of magnitude more and effectively eliminates the feature from your budget. Check your update cadence before enabling it. For static corpora (legal archives, product documentation that updates quarterly), it is excellent. For live news feeds or streaming data, the caching assumption breaks.

Late chunking at document length. jina-embeddings-v3 has an 8,192-token context window. A 50-page PDF is ~25,000 tokens. Late chunking on that document truncates at 8,192 tokens and applies standard embedding to the remainder — with no warning. You need to measure whether your corpus fits within the embedding model's context window and route oversized documents appropriately (hierarchical summarization, or standard chunking with explicit section headers as context prefix).

Chunk size tuned on the wrong corpus. A 512-token chunk is the right default for general-purpose English text. It is wrong for dense mathematical proofs (where 100 tokens may be a complete proof step), for conversational transcripts (where a speaker turn might be 20 tokens), and for code (where function length varies by two orders of magnitude). Route by document type before applying chunk size uniformly.

Chunks longer than the embedder's max sequence. Many widely used embedding models cap input at 512 tokens — most sentence-transformers models, plus several hosted embedders. Feed them the 1,024-token chunks from the opening story and everything past token 512 is silently dropped before embedding: no error, no warning, just a vector that represents the first half of each chunk. Retrieval for anything in the second half fails in a way that looks exactly like a relevance problem. Check model.max_seq_length (or the provider's documented limit) against your chunk size before ingesting, and make truncation loud — log it, don't swallow it.

Overlap is not free. A 64-token overlap on 400-token chunks inflates the index by ~19%: more vectors to store, embed, and search. Worse, overlapping chunks are near-duplicates, and near-duplicates cluster in embedding space — a query that matches the boundary region can fill three of your top-5 slots with variations of the same passage, crowding out genuinely distinct evidence. Overlap buys insurance against a fact straddling a boundary; it costs index size and result diversity. Keep it small (10-15% of chunk size), and if your top-k results look repetitive, deduplicate by parent document before handing chunks to the LLM.

Not measuring retrieval separately. The most common diagnostic error: low end-to-end answer quality gets attributed to the LLM when it is actually a retrieval failure — the right chunk was never in the top-k. Measure retrieval recall (did the relevant chunk appear in top-10?) independently of generation quality. If retrieval recall is 60%, fixing chunking will improve end-to-end quality more than any amount of prompt engineering. See RAG evaluation with Ragas for the metric framework.

Ignoring the lost-in-the-middle problem. Retrieval quality is only half the story. Even if the right chunks are retrieved, the LLM degrades 30%+ in accuracy when relevant content lands in the middle of a long context window. Ordering retrieved chunks so the highest-scored ones appear at the start and end of the context window is a cheap, effective fix that has nothing to do with chunking but is worth doing alongside any chunking improvement.

The decision in practice

Pick the simplest strategy that your retrieval metrics support, not the most sophisticated one.

For a new project with no baseline: start with recursive character splitting at 400 tokens with 64-token overlap. Get hybrid search (BM25 + dense retrieval) running and a cross-encoder reranker in place. Measure retrieval recall on a 50-question golden set before touching chunking again. This baseline gets you 80% of the quality available from much more complex approaches.

If retrieval recall on your golden set is above 80%, chunking is probably not your bottleneck — look at query transformation or reranker quality instead.

If recall is below 80% and your corpus is narrative text within 8K tokens per document: try late chunking. Run the same 50 questions and compare recall before and after.

If recall is below 80% and your corpus is structured reference documentation (legal, technical, medical): try parent-child chunking first. The structure-awareness usually helps more than embedding methodology changes for these document types.

If recall is above 80% but precision (are the top-5 retrieved chunks actually relevant?) is low: add Contextual Retrieval with prompt caching enabled. Budget ~$1.02 per million document tokens and treat it as a one-time ingestion cost for stable corpora.

The research shows late chunking + contextual retrieval as the quality ceiling for flat RAG. That is also the most expensive combination — both in engineering time and per-chunk cost. The right answer for your production system depends on what your retrieval metrics actually show, not on what performs best in published benchmarks on BEIR.

One thing the benchmarks do not tell you: a simpler chunking strategy with good hybrid search and a well-tuned reranker often outperforms a complex chunking strategy with poor retrieval configuration. Chunking matters, but it does not matter in isolation. The document ingestion pipeline — how documents are parsed and cleaned before chunking — is where the real unglamorous work happens, and poor parsing (garbled PDF text, missing table structure, stripped headers) makes any chunking strategy look bad. Chunking is the second decision, not the first.

{ "type": "lost-middle", "title": "Where retrieved chunks land matters as much as which ones you retrieve" }

If you are building a RAG system that needs to actually work in production — not just on your demo corpus — the evaluation loop described in RAG from scratch gives you the measurement infrastructure you need before making any of these choices. Measure first. Chunk second.