Document Ingestion Pipelines: The Unglamorous 80% of RAG

You shipped the RAG demo on a Friday. The retrieval looked great on the test set — manually curated questions with tidy answers. Two weeks later, your system is confidently telling users that the Q3 pricing table shows a 15% discount on enterprise contracts. The actual table shows 5%. Nobody touched the LLM. Nobody changed the prompt. The problem is that the PDF parser extracted the merged table cells in the wrong order, creating a corrupted text block that your chunker split at exactly the wrong boundary, producing a chunk that looks authoritative and retrieves perfectly for "enterprise discount" queries but contains hallucinated-looking numbers that were actually a parsing artifact.

This is the unglamorous 80% of RAG. Everything downstream of the parser — embedding model, retrieval strategy, reranking, generation — is operating on whatever the parser hands it. And in most production corpora, the parser is the weakest link.

Why parsing is the real problem

Text from a .txt or well-structured HTML file is easy. PDFs are not. PDFs are a presentation format, not a content format. They describe where to draw glyphs on a page; they do not encode semantic structure. A heading in a PDF is just a larger glyph at a specific coordinate. A table is a grid of text boxes that may or may not be aligned in a way that implies rows and columns. A two-column layout will, if naively extracted, interleave the left and right column text token by token, producing incoherent sentences.

The practical categories of PDF difficulty:

• Clean digital PDFs (born digital, no scans): text extraction works reliably with most parsers.
• PDFs with tables: table structure is notoriously hard. Cells that span multiple columns, merged headers, footnotes inside table cells — most parsers mangle these.
• Scanned PDFs: image-only; require OCR before any text extraction. OCR quality varies by scan quality, font, and language.
• Mixed PDFs: mostly digital text with some scanned pages or embedded images containing text (diagrams with labels, screenshots of code).
• Forms: structured field-value pairs that look nothing like body text.

The most dangerous class is documents that look fine to a naive parser but contain subtle errors — transposed table columns, missing page headers that provided context for everything below them, figures whose captions ended up merged with the preceding paragraph.

The three parsers worth knowing

Docling (IBM Research, open-source since late 2024) is purpose-built for complex enterprise documents. It reconstructs document layout from PDF rendering, identifies tables with a dedicated table recognition model, and outputs structured JSON with explicit chunk boundaries. For corpora heavy on PDFs with tables — legal documents, financial reports, regulatory filings — Docling's table extraction is significantly more reliable than text-extraction-only approaches. It is not fast: expect 200-500ms per page for complex documents on a modern CPU. GPU acceleration helps.

Marker (open-source) takes a different approach: it uses a vision model to convert PDF pages to clean Markdown. The output quality on clean PDFs is excellent. On scanned content it depends on the quality of the underlying OCR pass. Marker is faster than Docling on simpler documents. It handles mathematical notation better than most alternatives.

Unstructured takes a connector-first approach. It covers 25+ source types — PDF, DOCX, HTML, Confluence, Google Drive, Notion, S3, SharePoint — with a unified output format. For teams whose knowledge lives across many systems, this matters more than per-document parse quality. Unstructured's PDF quality is solid for clean documents; it falls behind Docling on complex table-heavy PDFs.

The honest recommendation: benchmark on your actual documents before committing. Take 20-30 representative samples from your corpus, parse them with each tool, and visually inspect the output. Table extraction quality in particular needs human eyes, not automated metrics. A single corrupted table in a 200-chunk document can produce three or four plausible-looking but factually wrong chunks.

# Docling: parse a PDF and get structured chunks
from docling.document_converter import DocumentConverter

converter = DocumentConverter()
result = converter.convert("path/to/document.pdf")

# Docling outputs DoclingDocument with explicit structure
doc = result.document

# Export as Markdown (preserves table formatting)
markdown_text = doc.export_to_markdown()

# Or chunk with Docling's structure-aware chunker
from docling.chunking import HybridChunker

chunker = HybridChunker(max_tokens=512)
for chunk in chunker.chunk(doc):
    print(chunk.text, chunk.meta)

# Unstructured: same interface across source types
from unstructured.partition.auto import partition

# Works identically for PDF, DOCX, HTML, email
elements = partition("path/to/document.pdf")

# Elements have types: Title, NarrativeText, Table, ListItem, etc.
text_elements = [e for e in elements if e.category in ("NarrativeText", "Table")]

Scanned and mixed corpora: OCR or a vision model

If a meaningful fraction of your corpus is scanned — 20-40% is typical for legal, healthcare, and insurance archives — the parser choice above is only half the decision. Docling and Marker both bundle an OCR pass (EasyOCR- and Tesseract-family engines) that engages automatically when a page has no extractable text layer, and for clean office scans at 300 DPI the bundled OCR is usually good enough. Validate it the same way you validate tables: parse 20 scanned samples and read the output. Where it falls apart is predictable — faxes, skewed or low-resolution scans, handwriting, stamps and annotations over text, non-Latin scripts.

The mid-2026 alternative for those pages is to skip classical OCR and send the page image to a vision model that emits Markdown directly. Output on messy scans is markedly better, and table structure comes out intact instead of as an afterthought. The trade-off is cost and throughput: a page image plus prompt runs roughly 1,000–1,500 tokens through a small multimodal model, which works out to somewhere around $0.50–2 per 1,000 pages (illustrative) versus effectively free CPU-based OCR — trivial for a 50,000-page corpus, real money at 50 million, and an order of magnitude slower per page than Docling. The pattern that works in practice is routing: let the cheap parser handle everything, flag pages with no text layer or that fail structural validation, and send only those to the vision model.

Metadata: the scaffolding you cannot retrofit

Metadata design is decided at ingestion time or not at all. Retrofitting metadata fields after the index is built requires a full re-ingest, and that re-ingest will be painful enough that teams defer it indefinitely, shipping with incomplete metadata and hoping retrieval quality compensates.

The mandatory fields:

• document_id: a stable, unique identifier for each source document. Not the file path (paths change), not the title (titles are not unique), not the URL (URLs change). A UUID or a hash of the canonical source identifier. Every chunk from the same document carries the same document_id. This is what makes incremental updates possible.
• chunk_index: the ordinal position of this chunk within its source document. With document_id and chunk_index you can reconstruct the document's chunk order, which matters for context window assembly.
• source_url / file_path: where the document lives. Used for provenance display ("Source: Q3 Pricing Policy, page 4") and for re-fetching the original during re-ingest.
• created_at: when the document was created or published. Not when it was ingested.
• updated_at: when the document was last modified. This is the field compared against the document registry to detect changes.
• document_type: pdf, html, docx, markdown, etc. Useful for type-specific retrieval filtering.
• access_level (or equivalent): who can see this document. If your RAG system serves users with different permission levels — admins vs. employees vs. contractors — access filtering must happen at retrieval time, not at prompt time. Pre-filtering by access level in the vector query is the correct pattern; relying on the LLM to "not mention" unauthorized content is not a security control.

Domain-specific metadata extends this core:

# Example chunk metadata schema
chunk_metadata = {
    "document_id": "doc_a3f9b2c1",
    "chunk_index": 7,
    "source_url": "s3://company-docs/legal/msa-template-v4.pdf",
    "created_at": "2025-11-15",
    "updated_at": "2026-04-02",
    "document_type": "pdf",
    "access_level": "internal",
    # domain-specific
    "department": "legal",
    "document_category": "contract_template",
    "effective_date": "2026-01-01",
    "language": "en",
}

Qdrant stores this as a payload alongside the vector, enabling filtered search: retrieve top-5 chunks where access_level == "internal" AND department == "legal". Weaviate and Pinecone support equivalent payload filtering.

flowchart TD
    DOC["Source document"] --> PARSE["Parse"]
    PARSE --> META_EXT["Extract metadata\n(document_id, dates, type)"]
    META_EXT --> DOMAIN["Apply domain labels\n(department, category)"]
    DOMAIN --> ACCESS["Set access_level\n(from source system ACL)"]
    ACCESS --> CHUNKS["Chunk → embed → store\nwith full metadata payload"]
    CHUNKS --> QUERY["At query time:\nfilter by access_level\nbefore ANN search"]
    style ACCESS fill:#ff2e88,color:#111
    style QUERY fill:#0e7490,color:#fff

The access control point is worth emphasizing. Filtering in the ANN query (pre-filtering) is faster and safer than filtering on returned results (post-filtering). With post-filtering you retrieve top-k from the full index and then discard unauthorized results — if most results were unauthorized, you may return fewer than k chunks or none at all, silently degrading retrieval quality. Pre-filtering narrows the search space first, then runs ANN within that subset.

PII scrubbing: before the embedding model, not after

The order matters. Once a chunk is embedded, the PII is encoded into the vector. It is retrievable. If a user queries for "contact information for John Smith", the vector index will happily surface a chunk containing John Smith's phone number and address if that chunk was embedded without scrubbing. Output filters on the LLM are not a defense: they miss structured PII reliably, they are not auditable, and they do nothing about direct vector retrieval in hybrid search pipelines.

Scrubbing goes between parsing and chunking, operating on the raw extracted text.

Microsoft Presidio (open-source) is the production standard. It combines NER models, regex patterns, and checksums to detect and redact a configurable list of entity types: names, email addresses, phone numbers, SSNs, credit card numbers, IP addresses, dates of birth, and domain-specific patterns you define. It runs as a Python library or a REST service.

from presidio_analyzer import AnalyzerEngine
from presidio_anonymizer import AnonymizerEngine

analyzer = AnalyzerEngine()
anonymizer = AnonymizerEngine()

def scrub_pii(text: str, language: str = "en") -> str:
    results = analyzer.analyze(
        text=text,
        language=language,
        # specify entity types or leave empty for all defaults
        entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN", "CREDIT_CARD"],
    )
    anonymized = anonymizer.anonymize(text=text, analyzer_results=results)
    return anonymized.text

raw_text = "Contact Sarah Chen at sarah.chen@example.com or call 555-867-5309."
clean_text = scrub_pii(raw_text)
# → "Contact <PERSON> at <EMAIL_ADDRESS> or call <PHONE_NUMBER>."

Presidio's recall is not perfect — it will miss obfuscated PII ("s dot chen at example dot com"), PII in images (which your parser may or may not have OCR'd), and domain-specific identifiers it was not configured to recognize (internal employee IDs, proprietary account number formats). For high-compliance environments (healthcare, finance, legal), augment Presidio with domain-specific regex patterns and run a second-pass validation on a sample of scrubbed output.

One practical nuance: if you need reversibility — auditors who need to trace a retrieved chunk back to a specific individual — maintain a secure mapping from redaction tokens to original values, stored separately from the vector index with its own access controls. Most teams do not need this; most teams also do not think about it until an auditor asks.

Chunk-at-ingest vs chunk-at-query

This distinction comes up repeatedly in design discussions, and the answer is almost always: chunk at ingest.

Chunking at ingest time means you compute chunk boundaries once, store the chunks, and pay zero chunking cost at query time. Retrieval hits pre-chunked content directly. This is the standard RAG architecture.

Chunking at query time means the retriever receives raw documents (or large document segments) and applies chunking logic during the retrieval request. The only real use case is hierarchical retrieval: you want to retrieve at fine-grained granularity for precision but return coarser-grained context to the LLM for coherence. The solution is not chunking at query time — it is storing both at ingest time.

Hierarchical (parent-child) chunking is the correct pattern:

# At ingest: store parent chunks and child chunks with linking
def ingest_hierarchical(document_text: str, doc_id: str):
    # Large parent chunks (800-1200 tokens) for context
    parent_chunks = split_into_chunks(document_text, size=1000, overlap=100)
    
    for parent_idx, parent in enumerate(parent_chunks):
        parent_chunk_id = f"{doc_id}_parent_{parent_idx}"
        
        # Small child chunks (200-300 tokens) for precision retrieval
        child_chunks = split_into_chunks(parent.text, size=250, overlap=25)
        
        for child_idx, child in enumerate(child_chunks):
            child_chunk_id = f"{doc_id}_parent_{parent_idx}_child_{child_idx}"
            child_metadata = {
                **base_metadata,
                "chunk_id": child_chunk_id,
                "parent_chunk_id": parent_chunk_id,  # link to parent
                "chunk_level": "child",
            }
            embed_and_store(child.text, child_chunk_id, child_metadata)
        
        # Also store parent (optionally embed it too for parent-level retrieval)
        store_parent_chunk(parent.text, parent_chunk_id, doc_id)

# At query time: retrieve child chunks, fetch parent for context
def retrieve_with_parent_context(query: str, top_k: int = 5):
    child_results = vector_search(query, filter={"chunk_level": "child"}, top_k=top_k)
    
    # Fetch the parent chunk for each result
    contexts = []
    for child in child_results:
        parent = fetch_parent_chunk(child.metadata["parent_chunk_id"])
        contexts.append(parent.text)  # Return parent context to LLM
    
    return contexts

This costs more storage (both levels are stored) but avoids recomputing chunk boundaries at query time and gives you the precision of small-chunk retrieval with the coherence of large-chunk context.

For a deeper treatment of chunking strategies themselves — semantic chunking, late chunking, contextual retrieval — see Chunking Strategies That Actually Matter. This article focuses on the ingestion pipeline mechanics.

Incremental index updates

A RAG system that re-ingests its entire corpus on every update is a toy. Production corpora change daily — documents are revised, new documents are added, old documents are deprecated. A full re-ingest of a 100,000-document corpus takes hours and costs real money in embedding API calls.

Incremental updates require three things:

1. A document registry: a table (can be a database, a simple JSON file, or a document in your vector store's metadata) mapping each document_id to its content_hash and last_ingested_at timestamp.
2. Stable chunk IDs: each chunk gets a deterministic ID derived from document_id and chunk_index (e.g., sha256(doc_id + str(chunk_index))). This makes it possible to delete exactly the chunks belonging to a changed document.
3. Filter-delete support in the vector store: Qdrant, Weaviate, Pinecone, and Milvus all support deletion by payload field value. delete where document_id == "doc_a3f9b2c1" removes all chunks from that document without touching anything else.

import hashlib
from datetime import datetime

def compute_content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

def run_incremental_ingest(source_documents: list[dict], registry: dict):
    """
    source_documents: list of {document_id, content, source_url, updated_at}
    registry: {document_id: {content_hash, last_ingested_at}}
    """
    to_ingest = []
    to_delete = []
    
    for doc in source_documents:
        doc_id = doc["document_id"]
        current_hash = compute_content_hash(doc["content"])
        
        if doc_id not in registry:
            # New document
            to_ingest.append(doc)
        elif registry[doc_id]["content_hash"] != current_hash:
            # Changed document — delete old chunks, re-ingest
            to_delete.append(doc_id)
            to_ingest.append(doc)
        # else: unchanged, skip
    
    # Delete stale chunks from changed documents
    for doc_id in to_delete:
        vector_store.delete(filter={"document_id": doc_id})
    
    # Ingest new and changed documents
    for doc in to_ingest:
        chunks = chunk_and_embed(doc)
        vector_store.upsert(chunks)
        registry[doc["document_id"]] = {
            "content_hash": compute_content_hash(doc["content"]),
            "last_ingested_at": datetime.utcnow().isoformat(),
        }
    
    return len(to_ingest), len(to_delete)

One thing the snippet glosses over: delete-then-insert is not atomic. If the process dies after vector_store.delete() but before the upsert completes — an embedding API failure is the usual culprit — the changed document is simply absent from the index, and every query during the gap silently misses it. The safer order is the reverse: insert the new chunks first (under new chunk IDs or a version tag in the payload), then delete the old ones, so the worst crash outcome is a brief window of duplicates rather than a hole. Either way, update the registry only after both operations succeed — then a rerun sees the stale hash and repairs the document instead of skipping it.

For documents with updated_at timestamps from the source system, you can use timestamp comparison instead of content hashing — faster, but you trust the source system's timestamps. Content hashing is more reliable; it catches documents where metadata was updated without changing content (irrelevant re-ingest) or where content changed without a metadata update (missed re-ingest).

flowchart TD
    FETCH["Fetch source documents"] --> HASH["Compute content hash"]
    HASH --> REG_CHECK{"In registry?\nHash changed?"}
    REG_CHECK -->|"New document"| INGEST["Parse → scrub → chunk → embed → insert"]
    REG_CHECK -->|"Changed document"| DELETE["Delete chunks\nwhere document_id = X"]
    DELETE --> INGEST
    REG_CHECK -->|"Unchanged"| SKIP["Skip"]
    INGEST --> UPDATE_REG["Update registry\n(hash + timestamp)"]
    SKIP --> DONE["Done"]
    UPDATE_REG --> DONE
    style DELETE fill:#ff2e88,color:#111
    style INGEST fill:#00e5ff,color:#0a0a0f
    style REG_CHECK fill:#ffaa00,color:#0a0a0f

One subtlety: document deletion from the source system. If a document disappears from your source (a file deleted from SharePoint, a Confluence page archived), your pipeline needs to detect its absence and remove its chunks from the index. The pattern is to compare the set of document IDs seen in the current crawl against the registry. Any ID in the registry but missing from the current crawl is a deleted document — remove its chunks.

What breaks

Parse failures that look like retrieval failures

When users report wrong answers, the diagnosis process usually starts at retrieval — are the right chunks being returned? — and sometimes never reaches parsing. Parse failures are insidious because the corrupted text looks like legitimate content in the index. Monitoring should include spot-checking a sample of parsed documents for known structural features (expected section headings, table row counts) rather than only measuring retrieval metrics.

The table problem

Tables are the most common source of high-confidence wrong answers in enterprise RAG. A parser that extracts a pricing table as interleaved column text produces chunks that contain real numbers in real sentences but in the wrong relationships. The embedding model encodes these as plausible finance-domain content. They retrieve for financial queries. The LLM generates a confident, grammatically correct answer with wrong numbers.

Detection requires table-specific validation: parse a representative sample of table-heavy documents, convert the extracted table back to a structured format (pandas DataFrame if you can), and compare a few known values against the source. If values match, the parser handled that table correctly. This sounds tedious because it is.

PII in embeddings that pre-dates scrubbing

If you ran ingestion before adding PII scrubbing, the existing index contains unredacted content. The scrubber in the new pipeline does nothing for previously-ingested chunks. You must re-ingest every document — there is no way to redact a vector after the fact. For compliance timelines, this matters: "we added scrubbing" and "the index is now PII-free" are not the same statement unless accompanied by a full re-ingest.

Metadata drift

Metadata fields added after initial index construction are missing from old chunks. If you add an access_level field in version 2 of your ingestion pipeline, chunks ingested in version 1 have no access_level in their payload. A pre-filter on access_level == "internal" will exclude all version 1 chunks. This is either a security hole (if you flip to post-filtering as a workaround) or a retrieval gap. The only fix is re-ingestion of pre-existing chunks with updated metadata.

Source connector failures

Unstructured and similar connectors pull from external systems (Confluence, SharePoint, Google Drive) via APIs that have rate limits, authentication timeouts, and permission changes. A connector that silently returns 200 results when the index expects 1,000 will produce a quietly degraded corpus. Monitor source connector health: compare document count per source on each pipeline run, alert on drops above 5%, and log which documents were skipped and why.

flowchart LR
    CONNECTOR["Source connector"] -->|"healthy: 1,000 docs"| INGEST_OK["Normal ingest"]
    CONNECTOR -->|"auth expired: 0 docs\nrate-limited: 200 docs"| ALERT["Alert: document count\ndropped > 5%"]
    ALERT --> DEAD_LETTER["Dead-letter queue\n(retry after fix)"]
    style ALERT fill:#ff2e88,color:#111
    style DEAD_LETTER fill:#ffaa00,color:#0a0a0f

Embedding model changes

If you upgrade your embedding model mid-corpus — new model, new dimensionality, new normalization — your existing vectors are incompatible with new ones. Cosine similarity between a vector produced by text-embedding-3-small and one produced by text-embedding-3-large is meaningless. Any model upgrade requires a full re-ingest. Track the embedding_model and embedding_model_version in both chunk metadata and the document registry so you know exactly which chunks need to be regenerated.

The cost math

Full initial ingest of 50,000 documents (avg 30 pages, ~15,000 tokens/doc):
  Parse (Docling, 32 workers, ~200ms/page avg): 1.5M pages → ~2.6 hours
  Total tokens to embed: 50,000 × 15,000 = 750M tokens
    At text-embedding-3-small ($0.02/1M tokens): $15.00
    At text-embedding-3-large ($0.13/1M tokens): $97.50
  Storage: 750,000 chunks × 512-dim float32 = 1.5 GB vectors + metadata

Daily incremental (3% of corpus changes = 1,500 docs):
  Parse: 45,000 pages → ~5 minutes
  Embed: 1,500 × 15,000 = 22.5M tokens → $0.45 at small-model pricing
  Index ops (delete + insert ~22,500 chunks): <30 seconds in Qdrant

Contextual Retrieval enrichment (Anthropic, Sept 2024):
  Add LLM-generated 50-100 token context prefix to each chunk at ingest time
  Cost: ~$1.02 per million document tokens with prompt caching enabled
  → Full corpus: ~$765 at 750M tokens; daily delta: ~$23
  Retrieval failure reduction: ~49% standalone, ~67% with reranking

The Contextual Retrieval enrichment math deserves attention. Generating a context prefix for every chunk via an LLM sounds expensive, but with Anthropic prompt caching the cost is around $1.02 per million document tokens — the system prompt and most of the document content hit the cache on the per-chunk calls. Without prompt caching, the cost is an order of magnitude higher and the feature becomes impractical at scale. See Chunking Strategies That Actually Matter for the technique; the key operational point here is that enrichment must be designed into the ingestion pipeline, not bolted on afterward.

The visualizer: the full pipeline in motion

{ "type": "rag-pipeline", "query": "policy", "title": "Ingestion and retrieval: the full pipeline" }

When to use what

The ingestion pipeline is not one thing. It is a set of decisions that compound:

Choose Docling when your corpus is complex PDFs — insurance claim forms, contracts, technical manuals dense with tables. Pay the parse time cost upfront; you pay it once per document. Choose Unstructured when your knowledge base spans many source systems and connector breadth matters more than parse quality on any single document type. Use Marker for clean PDF-dominant corpora where speed matters and tables are mostly decorative.

PII scrubbing and metadata design share a property: both are decided before the first embedding call or paid for later at full re-ingest cost. Scrub before embedding even for internal-only systems — access controls get misconfigured, index data gets exported, and the cost of scrubbing is low while the cost of a compliance incident is not. The same one-way door applies to metadata: the fields you will want six months in — access levels, department labels, document categories — are cheap to add now and expensive to retrofit. When in doubt, add the field and leave it empty rather than omitting it.

Incremental updates earn their keep from day one if your corpus changes more than weekly. The document registry is a small amount of code and saves the full re-ingest cost every time you run. A corpus that costs $15 to embed initially costs $0.45/day incrementally — the registry pays for itself on the second daily run.

And validate parse output on real documents before committing to a tool. This is the recommendation that teams skip most often and regret most reliably. Parse a representative 50-document sample with your top two candidates, read the output for table sections specifically, and make the decision based on what you see. The 30 minutes spent here prevents weeks of debugging wrong answers in production.

For everything that happens after ingestion — what retrieval strategy to run against the index, how to combine BM25 with dense search, how to evaluate what you built — see RAG from Scratch, Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking, and RAG Evaluation, Failure Modes, and the Decision Between RAG, Long Context, and Fine-Tuning. The ingestion pipeline covered here is the foundation; it does not matter what you build on top of it if the foundation is corrupt.