# Document Ingestion Pipelines: The Unglamorous 80% of RAG > PDF parsing, metadata design, PII scrubbing, chunk-at-ingest vs chunk-at-query, and incremental index updates — the ingestion work that determines whether RAG succeeds or fails. Canonical: https://ironclad.academy/ai-engineering/articles/document-ingestion-pipelines-rag | Difficulty: medium | Topics: RAG, Data Engineering, Vector Search, Privacy ## Key takeaways - Parse quality is upstream of everything — a bad PDF parser introduces errors that no amount of clever chunking or embedding can recover from. - Chunk at ingest, not at query time; hierarchical RAG is not an exception — it stores parent and child chunks at ingest and only selects the granularity to return at query time. - PII scrubbing must happen before embedding — once PII is encoded into a vector, it is retrievable and auditing becomes impossible. - Incremental index updates require a document registry with content hashes; without it you either re-ingest the entire corpus on every run or silently serve stale chunks. - Metadata design is not an afterthought — document_id, access_level, and timestamps are mandatory from day one; retrofitting them after index construction requires a full rebuild. - Table and figure extraction is the hardest parsing problem and the most common source of hallucination in enterprise RAG — validate parser output on your actual documents before committing to a tool. > **In one line:** The ingestion pipeline — parse, enrich, chunk, embed, index — is where most RAG failures originate, and it gets a fraction of the design attention that retrieval and generation do. **The idea.** Most RAG articles start at the embedding model and treat the document as already-clean text. In production, the document is a 200-page PDF with tables spanning multiple pages, scanned pages that fool text extractors, headers and footers that corrupt chunk boundaries, and PII that cannot legally enter a vector index. The ingestion pipeline handles all of this before retrieval ever happens. Get it wrong and retrieval failures are guaranteed — wrong chunks are the root cause behind most of what teams misdiagnose as LLM hallucination. **Key points.** - Parse quality gates everything downstream. The best embedding model cannot recover from corrupted text or missed tables. - Metadata must be designed up front: document_id, access_level, timestamps, and domain labels are the foundation for incremental updates and authorization filtering at retrieval time. - PII scrubbing belongs between parsing and embedding — never after. Vectors encode what you give them. - Chunking strategy (covered in depth in [Chunking Strategies That Actually Matter](/ai-engineering/articles/chunking-strategies-fixed-semantic-late)) is the highest-variance decision; the ingestion pipeline must support whatever strategy you choose without re-parsing. - Incremental updates require a document registry. Without one, every corpus change means a full re-ingest. **Ingestion pipeline (back-of-the-envelope).** ``` Corpus: 10,000 PDFs, avg 50 pages each = 500,000 pages Parse: ~100–300ms per page with Docling (simple → complex layouts) = 500,000 × 0.2s (mid estimate) = 100,000s ≈ 28 hours single-threaded With 32 workers: ~53 minutes Chunk + embed (assuming avg 500 tokens/page, 1.5 chunks/page): 750,000 chunks × 512 dims × float32 = ~1.5 GB raw vectors Embedding cost at text-embedding-3-small ($0.02/million tokens): 750,000 chunks × ~250 tokens/chunk = 187.5M tokens → ~$3.75 Incremental run (10% of corpus changed daily = 1,000 docs): Parse + embed: ~5 minutes with same worker pool (using ~200ms/page avg) Index update: delete + insert ~75,000 chunks → seconds in Qdrant/Weaviate ``` **The pipeline in one diagram.** ```mermaid flowchart LR SRC["Source documents\n(PDF, DOCX, HTML, S3)"] --> PARSE["Parser\n(Docling / Marker / Unstructured)"] PARSE --> META["Metadata extraction\n(title, date, access_level)"] META --> PII["PII scrubbing\n(Presidio / spaCy NER)"] PII --> CHUNK["Chunker\n(fixed / semantic / late)"] CHUNK --> ENRICH["Optional enrichment\n(contextual prefix, table → text)"] ENRICH --> EMBED["Embedding model"] EMBED --> INDEX["Vector index\n+ metadata store"] INDEX --> REGISTRY["Document registry\n(content hash + chunk IDs)"] style PARSE fill:#00e5ff,color:#0a0a0f style PII fill:#ff2e88,color:#111 style INDEX fill:#0e7490,color:#fff style REGISTRY fill:#15803d,color:#fff ``` **Key design decisions.** | Decision | Recommended default | When to deviate | | --- | --- | --- | | Parser | Docling for complex PDFs; Unstructured for source diversity | Marker if corpus is clean PDFs and parse speed matters | | PII scrubbing | Presidio before embedding | Skip only if corpus is provably PII-free (e.g., public web pages) | | Chunk timing | At ingest | Never chunk raw docs at query time; for hierarchical retrieval store parent + child chunks at ingest | | Metadata fields | document_id, source_url, created_at, updated_at, access_level | Add domain labels (department, product) as needed | | Incremental updates | Content-hash registry + filter-delete by document_id | Full re-ingest only if corpus is small (<10K docs) and cheap to embed | | Table handling | Extract as structured text; store as separate chunks | Ignore only if tables are decorative and contain no facts | **If you have 60 seconds, say this.** "Ingestion is parse, enrich, chunk, embed, index. The parser choice matters enormously — Docling for complex PDFs with tables, Unstructured if you're pulling from fifteen different sources. PII scrubbing goes between parsing and embedding, never after. I design metadata from day one: document ID, access level, timestamps — without those I can't do incremental updates or authorization filtering at retrieval time. Chunking happens at ingest, not at query time — even hierarchical setups chunk at ingest, storing parent and child chunks separately and picking the granularity at query time. For updates, I run a document registry with content hashes — on each pipeline run I only re-ingest what changed, which keeps the daily job down to minutes instead of hours." You shipped the RAG demo on a Friday. The retrieval looked great on the test set — manually curated questions with tidy answers. Two weeks later, your system is confidently telling users that the Q3 pricing table shows a 15% discount on enterprise contracts. The actual table shows 5%. Nobody touched the LLM. Nobody changed the prompt. The problem is that the PDF parser extracted the merged table cells in the wrong order, creating a corrupted text block that your chunker split at exactly the wrong boundary, producing a chunk that looks authoritative and retrieves perfectly for "enterprise discount" queries but contains hallucinated-looking numbers that were actually a parsing artifact. This is the unglamorous 80% of RAG. Everything downstream of the parser — embedding model, retrieval strategy, reranking, generation — is operating on whatever the parser hands it. And in most production corpora, the parser is the weakest link. ## Why parsing is the real problem Text from a `.txt` or well-structured HTML file is easy. PDFs are not. PDFs are a presentation format, not a content format. They describe where to draw glyphs on a page; they do not encode semantic structure. A heading in a PDF is just a larger glyph at a specific coordinate. A table is a grid of text boxes that may or may not be aligned in a way that implies rows and columns. A two-column layout will, if naively extracted, interleave the left and right column text token by token, producing incoherent sentences. The practical categories of PDF difficulty: - **Clean digital PDFs** (born digital, no scans): text extraction works reliably with most parsers. - **PDFs with tables**: table structure is notoriously hard. Cells that span multiple columns, merged headers, footnotes inside table cells — most parsers mangle these. - **Scanned PDFs**: image-only; require OCR before any text extraction. OCR quality varies by scan quality, font, and language. - **Mixed PDFs**: mostly digital text with some scanned pages or embedded images containing text (diagrams with labels, screenshots of code). - **Forms**: structured field-value pairs that look nothing like body text. The most dangerous class is documents that *look* fine to a naive parser but contain subtle errors — transposed table columns, missing page headers that provided context for everything below them, figures whose captions ended up merged with the preceding paragraph. ### The three parsers worth knowing **Docling** (IBM Research, open-source since late 2024) is purpose-built for complex enterprise documents. It reconstructs document layout from PDF rendering, identifies tables with a dedicated table recognition model, and outputs structured JSON with explicit chunk boundaries. For corpora heavy on PDFs with tables — legal documents, financial reports, regulatory filings — Docling's table extraction is significantly more reliable than text-extraction-only approaches. It is not fast: expect 200-500ms per page for complex documents on a modern CPU. GPU acceleration helps. **Marker** (open-source) takes a different approach: it uses a vision model to convert PDF pages to clean Markdown. The output quality on clean PDFs is excellent. On scanned content it depends on the quality of the underlying OCR pass. Marker is faster than Docling on simpler documents. It handles mathematical notation better than most alternatives. **Unstructured** takes a connector-first approach. It covers 25+ source types — PDF, DOCX, HTML, Confluence, Google Drive, Notion, S3, SharePoint — with a unified output format. For teams whose knowledge lives across many systems, this matters more than per-document parse quality. Unstructured's PDF quality is solid for clean documents; it falls behind Docling on complex table-heavy PDFs. The honest recommendation: **benchmark on your actual documents before committing**. Take 20-30 representative samples from your corpus, parse them with each tool, and visually inspect the output. Table extraction quality in particular needs human eyes, not automated metrics. A single corrupted table in a 200-chunk document can produce three or four plausible-looking but factually wrong chunks. ```python # Docling: parse a PDF and get structured chunks from docling.document_converter import DocumentConverter converter = DocumentConverter() result = converter.convert("path/to/document.pdf") # Docling outputs DoclingDocument with explicit structure doc = result.document # Export as Markdown (preserves table formatting) markdown_text = doc.export_to_markdown() # Or chunk with Docling's structure-aware chunker from docling.chunking import HybridChunker chunker = HybridChunker(max_tokens=512) for chunk in chunker.chunk(doc): print(chunk.text, chunk.meta) ``` ```python # Unstructured: same interface across source types from unstructured.partition.auto import partition # Works identically for PDF, DOCX, HTML, email elements = partition("path/to/document.pdf") # Elements have types: Title, NarrativeText, Table, ListItem, etc. text_elements = [e for e in elements if e.category in ("NarrativeText", "Table")] ``` ### Scanned and mixed corpora: OCR or a vision model If a meaningful fraction of your corpus is scanned — 20-40% is typical for legal, healthcare, and insurance archives — the parser choice above is only half the decision. Docling and Marker both bundle an OCR pass (EasyOCR- and Tesseract-family engines) that engages automatically when a page has no extractable text layer, and for clean office scans at 300 DPI the bundled OCR is usually good enough. Validate it the same way you validate tables: parse 20 scanned samples and read the output. Where it falls apart is predictable — faxes, skewed or low-resolution scans, handwriting, stamps and annotations over text, non-Latin scripts. The mid-2026 alternative for those pages is to skip classical OCR and send the page image to a vision model that emits Markdown directly. Output on messy scans is markedly better, and table structure comes out intact instead of as an afterthought. The trade-off is cost and throughput: a page image plus prompt runs roughly 1,000–1,500 tokens through a small multimodal model, which works out to somewhere around $0.50–2 per 1,000 pages (illustrative) versus effectively free CPU-based OCR — trivial for a 50,000-page corpus, real money at 50 million, and an order of magnitude slower per page than Docling. The pattern that works in practice is routing: let the cheap parser handle everything, flag pages with no text layer or that fail structural validation, and send only those to the vision model. ## Metadata: the scaffolding you cannot retrofit Metadata design is decided at ingestion time or not at all. Retrofitting metadata fields after the index is built requires a full re-ingest, and that re-ingest will be painful enough that teams defer it indefinitely, shipping with incomplete metadata and hoping retrieval quality compensates. The mandatory fields: - **`document_id`**: a stable, unique identifier for each source document. Not the file path (paths change), not the title (titles are not unique), not the URL (URLs change). A UUID or a hash of the canonical source identifier. Every chunk from the same document carries the same `document_id`. This is what makes incremental updates possible. - **`chunk_index`**: the ordinal position of this chunk within its source document. With `document_id` and `chunk_index` you can reconstruct the document's chunk order, which matters for context window assembly. - **`source_url` / `file_path`**: where the document lives. Used for provenance display ("Source: Q3 Pricing Policy, page 4") and for re-fetching the original during re-ingest. - **`created_at`**: when the document was created or published. Not when it was ingested. - **`updated_at`**: when the document was last modified. This is the field compared against the document registry to detect changes. - **`document_type`**: pdf, html, docx, markdown, etc. Useful for type-specific retrieval filtering. - **`access_level`** (or equivalent): who can see this document. If your RAG system serves users with different permission levels — admins vs. employees vs. contractors — access filtering must happen at retrieval time, not at prompt time. Pre-filtering by access level in the vector query is the correct pattern; relying on the LLM to "not mention" unauthorized content is not a security control. Domain-specific metadata extends this core: ```python # Example chunk metadata schema chunk_metadata = { "document_id": "doc_a3f9b2c1", "chunk_index": 7, "source_url": "s3://company-docs/legal/msa-template-v4.pdf", "created_at": "2025-11-15", "updated_at": "2026-04-02", "document_type": "pdf", "access_level": "internal", # domain-specific "department": "legal", "document_category": "contract_template", "effective_date": "2026-01-01", "language": "en", } ``` Qdrant stores this as a payload alongside the vector, enabling filtered search: `retrieve top-5 chunks where access_level == "internal" AND department == "legal"`. Weaviate and Pinecone support equivalent payload filtering. ```mermaid flowchart TD DOC["Source document"] --> PARSE["Parse"] PARSE --> META_EXT["Extract metadata\n(document_id, dates, type)"] META_EXT --> DOMAIN["Apply domain labels\n(department, category)"] DOMAIN --> ACCESS["Set access_level\n(from source system ACL)"] ACCESS --> CHUNKS["Chunk → embed → store\nwith full metadata payload"] CHUNKS --> QUERY["At query time:\nfilter by access_level\nbefore ANN search"] style ACCESS fill:#ff2e88,color:#111 style QUERY fill:#0e7490,color:#fff ``` The access control point is worth emphasizing. Filtering in the ANN query (`pre-filtering`) is faster and safer than filtering on returned results (`post-filtering`). With post-filtering you retrieve `top-k` from the full index and then discard unauthorized results — if most results were unauthorized, you may return fewer than `k` chunks or none at all, silently degrading retrieval quality. Pre-filtering narrows the search space first, then runs ANN within that subset. ## PII scrubbing: before the embedding model, not after The order matters. Once a chunk is embedded, the PII is encoded into the vector. It is retrievable. If a user queries for "contact information for John Smith", the vector index will happily surface a chunk containing John Smith's phone number and address if that chunk was embedded without scrubbing. Output filters on the LLM are not a defense: they miss structured PII reliably, they are not auditable, and they do nothing about direct vector retrieval in hybrid search pipelines. Scrubbing goes between parsing and chunking, operating on the raw extracted text. **Microsoft Presidio** (open-source) is the production standard. It combines NER models, regex patterns, and checksums to detect and redact a configurable list of entity types: names, email addresses, phone numbers, SSNs, credit card numbers, IP addresses, dates of birth, and domain-specific patterns you define. It runs as a Python library or a REST service. ```python from presidio_analyzer import AnalyzerEngine from presidio_anonymizer import AnonymizerEngine analyzer = AnalyzerEngine() anonymizer = AnonymizerEngine() def scrub_pii(text: str, language: str = "en") -> str: results = analyzer.analyze( text=text, language=language, # specify entity types or leave empty for all defaults entities=["PERSON", "EMAIL_ADDRESS", "PHONE_NUMBER", "US_SSN", "CREDIT_CARD"], ) anonymized = anonymizer.anonymize(text=text, analyzer_results=results) return anonymized.text raw_text = "Contact Sarah Chen at sarah.chen@example.com or call 555-867-5309." clean_text = scrub_pii(raw_text) # → "Contact at or call ." ``` Presidio's recall is not perfect — it will miss obfuscated PII ("s dot chen at example dot com"), PII in images (which your parser may or may not have OCR'd), and domain-specific identifiers it was not configured to recognize (internal employee IDs, proprietary account number formats). For high-compliance environments (healthcare, finance, legal), augment Presidio with domain-specific regex patterns and run a second-pass validation on a sample of scrubbed output. One practical nuance: if you need reversibility — auditors who need to trace a retrieved chunk back to a specific individual — maintain a secure mapping from redaction tokens to original values, stored separately from the vector index with its own access controls. Most teams do not need this; most teams also do not think about it until an auditor asks. ## Chunk-at-ingest vs chunk-at-query This distinction comes up repeatedly in design discussions, and the answer is almost always: chunk at ingest. Chunking at ingest time means you compute chunk boundaries once, store the chunks, and pay zero chunking cost at query time. Retrieval hits pre-chunked content directly. This is the standard RAG architecture. Chunking at query time means the retriever receives raw documents (or large document segments) and applies chunking logic during the retrieval request. The only real use case is **hierarchical retrieval**: you want to retrieve at fine-grained granularity for precision but return coarser-grained context to the LLM for coherence. The solution is not chunking at query time — it is storing both at ingest time. **Hierarchical (parent-child) chunking** is the correct pattern: ```python # At ingest: store parent chunks and child chunks with linking def ingest_hierarchical(document_text: str, doc_id: str): # Large parent chunks (800-1200 tokens) for context parent_chunks = split_into_chunks(document_text, size=1000, overlap=100) for parent_idx, parent in enumerate(parent_chunks): parent_chunk_id = f"{doc_id}_parent_{parent_idx}" # Small child chunks (200-300 tokens) for precision retrieval child_chunks = split_into_chunks(parent.text, size=250, overlap=25) for child_idx, child in enumerate(child_chunks): child_chunk_id = f"{doc_id}_parent_{parent_idx}_child_{child_idx}" child_metadata = { **base_metadata, "chunk_id": child_chunk_id, "parent_chunk_id": parent_chunk_id, # link to parent "chunk_level": "child", } embed_and_store(child.text, child_chunk_id, child_metadata) # Also store parent (optionally embed it too for parent-level retrieval) store_parent_chunk(parent.text, parent_chunk_id, doc_id) # At query time: retrieve child chunks, fetch parent for context def retrieve_with_parent_context(query: str, top_k: int = 5): child_results = vector_search(query, filter={"chunk_level": "child"}, top_k=top_k) # Fetch the parent chunk for each result contexts = [] for child in child_results: parent = fetch_parent_chunk(child.metadata["parent_chunk_id"]) contexts.append(parent.text) # Return parent context to LLM return contexts ``` This costs more storage (both levels are stored) but avoids recomputing chunk boundaries at query time and gives you the precision of small-chunk retrieval with the coherence of large-chunk context. For a deeper treatment of chunking strategies themselves — semantic chunking, late chunking, contextual retrieval — see [Chunking Strategies That Actually Matter](/ai-engineering/articles/chunking-strategies-fixed-semantic-late). This article focuses on the ingestion pipeline mechanics. ## Incremental index updates A RAG system that re-ingests its entire corpus on every update is a toy. Production corpora change daily — documents are revised, new documents are added, old documents are deprecated. A full re-ingest of a 100,000-document corpus takes hours and costs real money in embedding API calls. Incremental updates require three things: 1. **A document registry**: a table (can be a database, a simple JSON file, or a document in your vector store's metadata) mapping each `document_id` to its `content_hash` and `last_ingested_at` timestamp. 2. **Stable chunk IDs**: each chunk gets a deterministic ID derived from `document_id` and `chunk_index` (e.g., `sha256(doc_id + str(chunk_index))`). This makes it possible to delete exactly the chunks belonging to a changed document. 3. **Filter-delete support in the vector store**: Qdrant, Weaviate, Pinecone, and Milvus all support deletion by payload field value. `delete where document_id == "doc_a3f9b2c1"` removes all chunks from that document without touching anything else. ```python import hashlib from datetime import datetime def compute_content_hash(text: str) -> str: return hashlib.sha256(text.encode()).hexdigest() def run_incremental_ingest(source_documents: list[dict], registry: dict): """ source_documents: list of {document_id, content, source_url, updated_at} registry: {document_id: {content_hash, last_ingested_at}} """ to_ingest = [] to_delete = [] for doc in source_documents: doc_id = doc["document_id"] current_hash = compute_content_hash(doc["content"]) if doc_id not in registry: # New document to_ingest.append(doc) elif registry[doc_id]["content_hash"] != current_hash: # Changed document — delete old chunks, re-ingest to_delete.append(doc_id) to_ingest.append(doc) # else: unchanged, skip # Delete stale chunks from changed documents for doc_id in to_delete: vector_store.delete(filter={"document_id": doc_id}) # Ingest new and changed documents for doc in to_ingest: chunks = chunk_and_embed(doc) vector_store.upsert(chunks) registry[doc["document_id"]] = { "content_hash": compute_content_hash(doc["content"]), "last_ingested_at": datetime.utcnow().isoformat(), } return len(to_ingest), len(to_delete) ``` One thing the snippet glosses over: delete-then-insert is not atomic. If the process dies after `vector_store.delete()` but before the upsert completes — an embedding API failure is the usual culprit — the changed document is simply absent from the index, and every query during the gap silently misses it. The safer order is the reverse: insert the new chunks first (under new chunk IDs or a version tag in the payload), then delete the old ones, so the worst crash outcome is a brief window of duplicates rather than a hole. Either way, update the registry only after both operations succeed — then a rerun sees the stale hash and repairs the document instead of skipping it. For documents with `updated_at` timestamps from the source system, you can use timestamp comparison instead of content hashing — faster, but you trust the source system's timestamps. Content hashing is more reliable; it catches documents where metadata was updated without changing content (irrelevant re-ingest) or where content changed without a metadata update (missed re-ingest). ```mermaid flowchart TD FETCH["Fetch source documents"] --> HASH["Compute content hash"] HASH --> REG_CHECK{"In registry?\nHash changed?"} REG_CHECK -->|"New document"| INGEST["Parse → scrub → chunk → embed → insert"] REG_CHECK -->|"Changed document"| DELETE["Delete chunks\nwhere document_id = X"] DELETE --> INGEST REG_CHECK -->|"Unchanged"| SKIP["Skip"] INGEST --> UPDATE_REG["Update registry\n(hash + timestamp)"] SKIP --> DONE["Done"] UPDATE_REG --> DONE style DELETE fill:#ff2e88,color:#111 style INGEST fill:#00e5ff,color:#0a0a0f style REG_CHECK fill:#ffaa00,color:#0a0a0f ``` One subtlety: document deletion from the source system. If a document disappears from your source (a file deleted from SharePoint, a Confluence page archived), your pipeline needs to detect its absence and remove its chunks from the index. The pattern is to compare the set of document IDs seen in the current crawl against the registry. Any ID in the registry but missing from the current crawl is a deleted document — remove its chunks. ## What breaks ### Parse failures that look like retrieval failures When users report wrong answers, the diagnosis process usually starts at retrieval — are the right chunks being returned? — and sometimes never reaches parsing. Parse failures are insidious because the corrupted text looks like legitimate content in the index. Monitoring should include spot-checking a sample of parsed documents for known structural features (expected section headings, table row counts) rather than only measuring retrieval metrics. ### The table problem Tables are the most common source of high-confidence wrong answers in enterprise RAG. A parser that extracts a pricing table as interleaved column text produces chunks that contain real numbers in real sentences but in the wrong relationships. The embedding model encodes these as plausible finance-domain content. They retrieve for financial queries. The LLM generates a confident, grammatically correct answer with wrong numbers. Detection requires table-specific validation: parse a representative sample of table-heavy documents, convert the extracted table back to a structured format (pandas DataFrame if you can), and compare a few known values against the source. If values match, the parser handled that table correctly. This sounds tedious because it is. ### PII in embeddings that pre-dates scrubbing If you ran ingestion before adding PII scrubbing, the existing index contains unredacted content. The scrubber in the new pipeline does nothing for previously-ingested chunks. You must re-ingest every document — there is no way to redact a vector after the fact. For compliance timelines, this matters: "we added scrubbing" and "the index is now PII-free" are not the same statement unless accompanied by a full re-ingest. ### Metadata drift Metadata fields added after initial index construction are missing from old chunks. If you add an `access_level` field in version 2 of your ingestion pipeline, chunks ingested in version 1 have no `access_level` in their payload. A pre-filter on `access_level == "internal"` will exclude all version 1 chunks. This is either a security hole (if you flip to post-filtering as a workaround) or a retrieval gap. The only fix is re-ingestion of pre-existing chunks with updated metadata. ### Source connector failures Unstructured and similar connectors pull from external systems (Confluence, SharePoint, Google Drive) via APIs that have rate limits, authentication timeouts, and permission changes. A connector that silently returns 200 results when the index expects 1,000 will produce a quietly degraded corpus. Monitor source connector health: compare document count per source on each pipeline run, alert on drops above 5%, and log which documents were skipped and why. ```mermaid flowchart LR CONNECTOR["Source connector"] -->|"healthy: 1,000 docs"| INGEST_OK["Normal ingest"] CONNECTOR -->|"auth expired: 0 docs\nrate-limited: 200 docs"| ALERT["Alert: document count\ndropped > 5%"] ALERT --> DEAD_LETTER["Dead-letter queue\n(retry after fix)"] style ALERT fill:#ff2e88,color:#111 style DEAD_LETTER fill:#ffaa00,color:#0a0a0f ``` ### Embedding model changes If you upgrade your embedding model mid-corpus — new model, new dimensionality, new normalization — your existing vectors are incompatible with new ones. Cosine similarity between a vector produced by `text-embedding-3-small` and one produced by `text-embedding-3-large` is meaningless. Any model upgrade requires a full re-ingest. Track the `embedding_model` and `embedding_model_version` in both chunk metadata and the document registry so you know exactly which chunks need to be regenerated. ## The cost math ``` Full initial ingest of 50,000 documents (avg 30 pages, ~15,000 tokens/doc): Parse (Docling, 32 workers, ~200ms/page avg): 1.5M pages → ~2.6 hours Total tokens to embed: 50,000 × 15,000 = 750M tokens At text-embedding-3-small ($0.02/1M tokens): $15.00 At text-embedding-3-large ($0.13/1M tokens): $97.50 Storage: 750,000 chunks × 512-dim float32 = 1.5 GB vectors + metadata Daily incremental (3% of corpus changes = 1,500 docs): Parse: 45,000 pages → ~5 minutes Embed: 1,500 × 15,000 = 22.5M tokens → $0.45 at small-model pricing Index ops (delete + insert ~22,500 chunks): <30 seconds in Qdrant Contextual Retrieval enrichment (Anthropic, Sept 2024): Add LLM-generated 50-100 token context prefix to each chunk at ingest time Cost: ~$1.02 per million document tokens with prompt caching enabled → Full corpus: ~$765 at 750M tokens; daily delta: ~$23 Retrieval failure reduction: ~49% standalone, ~67% with reranking ``` The Contextual Retrieval enrichment math deserves attention. Generating a context prefix for every chunk via an LLM sounds expensive, but with Anthropic prompt caching the cost is around $1.02 per million document tokens — the system prompt and most of the document content hit the cache on the per-chunk calls. Without prompt caching, the cost is an order of magnitude higher and the feature becomes impractical at scale. See [Chunking Strategies That Actually Matter](/ai-engineering/articles/chunking-strategies-fixed-semantic-late) for the technique; the key operational point here is that enrichment must be designed into the ingestion pipeline, not bolted on afterward. ## The visualizer: the full pipeline in motion [Interactive visualizer on the original page: rag-pipeline — Ingestion and retrieval: the full pipeline] ## When to use what The ingestion pipeline is not one thing. It is a set of decisions that compound: **Choose Docling** when your corpus is complex PDFs — insurance claim forms, contracts, technical manuals dense with tables. Pay the parse time cost upfront; you pay it once per document. Choose Unstructured when your knowledge base spans many source systems and connector breadth matters more than parse quality on any single document type. Use Marker for clean PDF-dominant corpora where speed matters and tables are mostly decorative. PII scrubbing and metadata design share a property: both are decided before the first embedding call or paid for later at full re-ingest cost. Scrub before embedding even for internal-only systems — access controls get misconfigured, index data gets exported, and the cost of scrubbing is low while the cost of a compliance incident is not. The same one-way door applies to metadata: the fields you will want six months in — access levels, department labels, document categories — are cheap to add now and expensive to retrofit. When in doubt, add the field and leave it empty rather than omitting it. Incremental updates earn their keep from day one if your corpus changes more than weekly. The document registry is a small amount of code and saves the full re-ingest cost every time you run. A corpus that costs $15 to embed initially costs $0.45/day incrementally — the registry pays for itself on the second daily run. And validate parse output on real documents before committing to a tool. This is the recommendation that teams skip most often and regret most reliably. Parse a representative 50-document sample with your top two candidates, read the output for table sections specifically, and make the decision based on what you see. The 30 minutes spent here prevents weeks of debugging wrong answers in production. For everything that happens after ingestion — what retrieval strategy to run against the index, how to combine BM25 with dense search, how to evaluate what you built — see [RAG from Scratch](/ai-engineering/articles/rag-from-scratch-ingestion-retrieval-generation), [Hybrid Search: BM25 Plus Dense Retrieval Plus Reranking](/ai-engineering/articles/hybrid-search-bm25-dense-reranking), and [RAG Evaluation, Failure Modes, and the Decision Between RAG, Long Context, and Fine-Tuning](/ai-engineering/articles/rag-evaluation-failure-modes-vs-alternatives). The ingestion pipeline covered here is the foundation; it does not matter what you build on top of it if the foundation is corrupt. ## Frequently asked questions Q: What is a document ingestion pipeline in RAG? A: A document ingestion pipeline transforms raw source documents (PDFs, HTML, DOCX, database exports) into indexed, searchable chunks ready for retrieval. The stages are: parse the raw file into clean text and structure, extract and attach metadata, split the text into chunks, optionally enrich chunks with LLM-generated context, embed each chunk, and write to a vector index. Getting these stages right determines retrieval quality far more than the choice of LLM or embedding model. Q: Which PDF parser should I use for RAG — Docling, Marker, or Unstructured? A: Docling (IBM, open-source) handles complex layouts with accurate table extraction and native chunking output, making it the strongest choice for enterprise document corpora heavy on PDFs. Marker (open-source) is fast and accurate for clean PDFs but struggles with scanned images without OCR. Unstructured offers connectors to 25+ source types (Google Drive, Confluence, S3, SharePoint) and is the pragmatic choice when your source diversity matters more than per-document parse quality. For production, benchmark all three on a representative sample of your actual documents; parse quality varies dramatically by domain. Q: Should I chunk documents at ingest time or at query time? A: Chunk at ingest time — always. You pay the chunking cost once, store the chunks, and retrieve pre-chunked content at query time with zero latency overhead. Hierarchical RAG is not an exception: it chunks at two granularities (parent and child) at ingest, stores both, and only selects which granularity to return at query time. Chunking raw documents at query time adds 20-200ms per request and recomputes work already done. Q: How do I handle PII in documents before ingesting them into a vector index? A: PII scrubbing belongs in the ingestion pipeline, before chunks are embedded and stored. Use a purpose-built NER-based scrubber (Microsoft Presidio, spaCy with custom rules, or a fine-tuned classifier) to detect and redact names, email addresses, phone numbers, SSNs, and domain-specific identifiers like account numbers. Store a mapping of original-to-redacted tokens if downstream workflows need re-identification under controlled conditions. Never rely solely on LLM output filters — they miss structured PII reliably and are not auditable. Scrubbing must happen before embedding because the embedding model will encode PII into vector space where it can be retrieved. Q: How do I update a vector index without re-ingesting the entire corpus? A: Incremental updates require a document registry that tracks each source document by a stable ID plus a content hash or last-modified timestamp. On each pipeline run, compare incoming documents against the registry: insert new documents, re-ingest and replace changed documents (delete old chunks by document ID, insert new chunks), and tombstone deleted documents. Qdrant, Weaviate, and Pinecone all support payload-filtered deletion by a document_id field, making this straightforward. The key is using a deterministic chunk ID scheme (hash of document_id + chunk_index) so you can detect which chunks changed without scanning the entire index. Q: What metadata fields should every RAG chunk have? A: At minimum: document_id (stable identifier for the source document), chunk_index (position within document), source_url or file_path, created_at and updated_at timestamps, document_type (pdf, html, docx, etc.), and any domain-specific classification labels (department, product, region, access_level). This metadata serves three purposes: enables incremental updates (filter-delete by document_id), enables pre-retrieval filtering (only search documents user is authorized to see), and enables post-retrieval provenance (show the user where the answer came from).