Document AI and OCR in 2026: When to Use VLMs vs Classic Engines

A legal team shipped a document review system in 2024 using GPT-4o for all extraction. Six months later the invoice processing was accurate and the team was happy — but the monthly API bill had grown to $18,000 as they scaled from 5,000 to 80,000 pages/day. The same accuracy was achievable for under $400/month self-hosted. Nobody had done the math before choosing the architecture.

That is a real category of mistake. The other failure runs in the opposite direction: teams use Tesseract on scanned insurance claims with handwritten annotations, multi-column layouts, and rotated stamps, get 40% field accuracy, and conclude that "OCR doesn't work." It works fine on clean text. It was the wrong tool for that document type.

The decision between classical OCR engines and vision-language models is not a question of what is newer or more capable in general — it is a specific engineering tradeoff that changes based on document type, volume, and the downstream cost of extraction errors.

What classical OCR actually does

Classical OCR engines work in two stages. First, a detection phase finds regions in the image that contain text — bounding boxes around lines, words, or characters. Second, a recognition phase classifies each detected region into characters. Tesseract 5 uses LSTM-based recognition; PaddleOCR uses a detection-recognition pipeline that can handle rotated and irregular text.

flowchart LR
    IMG[Document image] --> PREPROCESS[Preprocessing\ndeskew, binarize, denoise]
    PREPROCESS --> DETECT[Text region detection\nbounding boxes]
    DETECT --> RECOGNIZE[Character recognition\nLSTM / CTC]
    RECOGNIZE --> TEXT[Raw text string]
    TEXT --> PARSE[Structural parser\nregex, rules, heuristics]
    PARSE --> SCHEMA[Structured output]

The pipeline has no understanding of the document. It does not know that the number after "Invoice Total:" is the total. It does not know that the text in the left column of a two-column layout belongs to a different field from the text in the right column. All of that has to be encoded in the post-processing parser — and writing those parsers is where classical OCR systems spend 80% of their engineering time.

This is why classical OCR's real weakness is not character recognition accuracy on clean text — it is structure understanding. On a clean, single-column printed English document, Tesseract's character error rate is below 1%. On a scanned tax form with mixed fonts, handwritten values, rotated stamps, and a grid table, that number climbs to 25–40% before any structural parsing errors.

What VLMs bring to document extraction

A vision-language model processes the document image as visual tokens and produces structured output in a single pass. It understands that a row of numbers under a header is a table. It knows that a cursive annotation in the margin is a note, not part of the main text. It can follow a JSON schema and fill fields based on layout semantics, not just positional text extraction.

sequenceDiagram
    participant ENG as Engineer
    participant VLM as VLM API
    participant VAL as Pydantic validator

    ENG->>VLM: image bytes + JSON schema + system prompt
    Note over VLM: Single inference pass<br/>understands layout + content
    VLM-->>ENG: JSON string with extracted fields
    ENG->>VAL: parse and validate
    VAL-->>ENG: typed Invoice object (or ValidationError)

The code to do this with the Anthropic API is short:

import anthropic
import base64
from pathlib import Path
from pydantic import BaseModel, field_validator

class InvoiceLineItem(BaseModel):
    description: str
    quantity: int
    unit_price: float
    total: float

class Invoice(BaseModel):
    invoice_number: str
    vendor: str
    date: str
    line_items: list[InvoiceLineItem]
    subtotal: float
    tax: float
    total: float

    @field_validator("total")
    @classmethod
    def total_must_be_positive(cls, v: float) -> float:
        if v <= 0:
            raise ValueError("total must be positive")
        return v

def extract_invoice(image_path: str) -> Invoice:
    client = anthropic.Anthropic()
    image_data = base64.standard_b64encode(
        Path(image_path).read_bytes()
    ).decode("utf-8")

    response = client.messages.create(
        model="claude-opus-4-6",
        max_tokens=2048,
        system="""You are a document extraction engine. Extract the invoice fields
into the exact JSON schema provided. If a field is missing from the document,
use null. Do not invent values. Output only valid JSON, no prose.""",
        messages=[
            {
                "role": "user",
                "content": [
                    {
                        "type": "image",
                        "source": {
                            "type": "base64",
                            "media_type": "image/png",
                            "data": image_data,
                        },
                    },
                    {
                        "type": "text",
                        "text": f"""Extract this invoice into this JSON schema:
{Invoice.model_json_schema()}

Return only the JSON object, no markdown fences.""",
                    },
                ],
            }
        ],
    )

    raw = response.content[0].text.strip()
    return Invoice.model_validate_json(raw)

The schema validation step is non-negotiable. The same image fed to the same model on two separate calls can produce "invoice_number" in one response and "invoiceNumber" in the next, or omit optional fields entirely. Pydantic's model_validate_json catches this immediately rather than letting malformed data propagate into your database.

The 2026 accuracy picture

The OCR Arena leaderboard (as of early 2026) has Gemini 3 Flash at the top for multi-page document extraction, followed closely by Claude Opus 4.6. But the aggregate leaderboard number masks a pattern that matters for production: frontier VLMs excel on varied, noisy, real-world documents, while specialized models dominate on narrow benchmarks for specific document types.

dots.ocr (1.7B parameters, 2026) is the clearest example. It supports 100+ languages and outperforms models 20x its size on standard document benchmarks. The mechanism is not architectural magic — it is training data specificity. Frontier models like GPT-4o and Claude are trained primarily on instruction-following and general reasoning data; dots.ocr is trained exclusively on documents. The same specialization pattern that makes Whisper-large better than general audio models at transcription applies here.

PaddleOCR-VL-1.5 (January 2026) claims 95% accuracy on authoritative document parsing benchmarks, surpassing top proprietary models on structured form extraction while running self-hosted.

The key caveat: OCR benchmarks are systematically gamed. Published leaderboards use clean, high-resolution printed documents — the conditions where all models perform well. Real-world production accuracy is 30–50% worse on scanned documents with noise, varying lighting, and mixed languages. Before committing to any model, benchmark on your actual document corpus at the quality level you will actually receive. This is not optional advice.

The DPI decision

PDF rendering is where many teams lose accuracy quietly. Converting a PDF page to an image for VLM input requires choosing a resolution, and that choice has compounding effects.

DPI comparison for an A4 page (210 × 297 mm):

150 DPI → 1240 × 1754 pixels → ~2.2M pixels
300 DPI → 2480 × 3508 pixels → ~8.7M pixels (4x larger)

GPT-4o high-detail resizes twice before tiling: first fit within 2048×2048,
then scale the shortest side down to 768px.

150 DPI A4 (1240×1754 → fits in 2048 → shortest side 1240 → 768×1086):
  768px / 512px = 1.5 → ceil = 2 tiles wide
  1086px / 512px = 2.1 → ceil = 3 tiles tall
  = 6 tiles × 170 tokens/tile + 85 base = ~1,105 tokens/page (illustrative)

300 DPI A4 (2480×3508 → scaled to 1449×2048 → shortest side 1449 → 768×1085):
  768px / 512px = 1.5 → ceil = 2 tiles wide
  1085px / 512px = 2.1 → ceil = 3 tiles tall
  = 6 tiles × 170 tokens/tile + 85 base = ~1,105 tokens/page (illustrative)

Note: both DPI settings land on identical tile counts under GPT-4o's
resize-then-tile policy — the 768px shortest-side step erases the raw pixel
difference entirely. For GPT-4o, 300 DPI costs you rendering time and storage,
not API tokens.

Accuracy gain: negligible for clean print. Meaningful for handwriting,
small fonts (<9pt), and documents with significant noise or degradation.

The practical rule: use 150 DPI for clean printed documents (invoices, contracts, receipts from digital sources). Use 300 DPI for scanned documents, handwriting, small fonts, and anything with significant noise. For GPT-4o the token cost is flat either way, so the only reason not to default to 300 DPI is rendering and storage overhead — providers that bill on raw pixel count are a different story, so check the pricing rules before assuming this generalizes.

The three extraction architectures

There are three distinct approaches, each with different cost, accuracy, and complexity profiles.

Architecture 1: Classical OCR + structural parser

Extract text with Tesseract or PaddleOCR, then parse structure with regular expressions, layout analysis, or a small extraction model. This is the cheapest path at scale and the most debuggable — you can inspect the intermediate text to diagnose failures. It breaks down when the document structure is irregular, when handwriting appears, or when table cell boundaries are complex.

Architecture 2: VLM direct extraction

Send the rendered page image to a VLM with a JSON schema. The model handles OCR and structure understanding in one pass. This is the right choice for complex layouts, tables, and documents with mixed content types. The cost is 50–200x higher than self-hosted classical OCR at scale.

Architecture 3: OCR-free end-to-end (Donut pattern)

An encoder-decoder transformer reads the image and produces structured text directly, with no intermediate OCR step. Donut is the canonical example. For fixed-form documents where you have 500+ training examples in the target format, this approach achieves the highest accuracy at reasonable cost — but it requires fine-tuning on your specific document type and has no generalization capability. It is a specialized weapon, not a general-purpose tool. Do not confuse it with PaddleOCR-VL-1.5, which is also end-to-end but ships as a general-purpose document VLM you run off the shelf — architecturally it skips the OCR step like Donut, but in the decision framework it belongs with dots.ocr in the specialized open-weight column.

flowchart TD
    subgraph ARCH1["Architecture 1: Classical OCR + parser"]
        A1_IMG[Image] --> A1_OCR[Tesseract / PaddleOCR]
        A1_OCR --> A1_TEXT[Raw text]
        A1_TEXT --> A1_PARSE[Rule-based parser]
        A1_PARSE --> A1_OUT[Structured output]
    end
    subgraph ARCH2["Architecture 2: VLM direct"]
        A2_IMG[Image] --> A2_VLM["VLM + schema\nGPT-4o / Gemini / Claude"]
        A2_VLM --> A2_OUT[JSON output]
    end
    subgraph ARCH3["Architecture 3: OCR-free"]
        A3_IMG[Image] --> A3_E2E["Fine-tuned encoder-decoder\nDonut-style"]
        A3_E2E --> A3_OUT[Structured output]
    end

The multi-page batching problem

Single-page extraction is solved. The hard part in production is multi-page documents: a 40-page contract, a batch of 200 invoices, a quarterly report with embedded charts and footnotes across 80 pages.

Naive approaches send each page as a separate API call. This works but misses cross-page context: a value in the footer of page 3 that references a definition on page 1, a table that spans pages 5 and 6 with the header only on page 5. Gemini's long context (1M tokens as of mid-2026) handles this by allowing the entire document in one call — but at real cost. At the per-page baseline from the summary block (~1,100 image tokens plus prompt and output), a 40-page document runs 50,000–90,000 tokens in one request, and providers that encode images at higher token densities push that well past 100,000.

The practical approach for most teams is a hybrid: extract page-by-page in parallel, then do a second consolidation pass that resolves cross-page references and fills gaps. For documents with known structure (annual reports, legal contracts with numbered sections), a table of contents extraction pass can first identify which pages contain which sections, then extract sections in targeted parallel calls.

import json
from concurrent.futures import ThreadPoolExecutor
import anthropic
import base64

def extract_pages_parallel(
    pages: list[bytes],
    schema: dict,
    max_workers: int = 10,
) -> list[dict]:
    """Extract structured data from pages in parallel."""
    client = anthropic.Anthropic()

    def extract_page(page_bytes: bytes) -> dict:
        image_b64 = base64.standard_b64encode(page_bytes).decode()
        response = client.messages.create(
            model="claude-opus-4-6",
            max_tokens=1024,
            messages=[{
                "role": "user",
                "content": [
                    {"type": "image", "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_b64,
                    }},
                    {"type": "text", "text": f"Extract into: {schema}\nReturn JSON only."},
                ],
            }],
        )
        return json.loads(response.content[0].text)

    with ThreadPoolExecutor(max_workers=max_workers) as ex:
        return list(ex.map(extract_page, pages))

Rate limits matter here. Anthropic and OpenAI both have per-minute token limits; a burst of 10 parallel single-page requests is only ~25,000–30,000 tokens, but sustain that parallelism at a few seconds per page and you are consuming 200,000–300,000 tokens/minute — above the default tier for most accounts. Build exponential backoff with jitter around the extract function, and monitor token-per-minute consumption, not just requests-per-minute.

What breaks

VLM-based document extraction has three failure modes that appear in every production deployment.

Hallucinated values. When the model cannot read text (due to degradation, low contrast, unusual fonts), it generates a plausible-looking value rather than returning null. A total that should be "$1,847.23" becomes "$1,874.23" — transposed digits, entirely fabricated. This is the most dangerous failure mode because it is silent: the JSON is valid, the schema passes, the value is in the right range. The only way to catch it is downstream validation (does this invoice total match the sum of line items? does this date predate the document creation?) or a confidence scoring layer.

Table cell boundary errors. Complex tables with merged cells, spanning headers, or nested subrows are the hardest layout case for VLMs. The model frequently misidentifies which row or column a value belongs to, producing structurally plausible but semantically wrong output. Gemini 3 Flash currently handles this best of the frontier APIs, but "best" still means 10–15% error rate on genuinely complex tables in real-world scans.

Non-deterministic JSON structure. The same image, the same prompt, the same model, on two separate API calls can return {"invoiceDate": "2026-01-15"} on one run and {"invoice_date": "2026-01-15"} on the next. Or the model might include an "items" array on one call and "line_items" on another. Always use model_validate_json against a strict Pydantic model, not json.loads followed by dict access. Validation errors give you a clear failure signal; silent dict access gives you a KeyError in production three weeks later.

The field is not solved. No VLM reliably handles all three failure modes. Build your pipeline assuming extraction will be imperfect, with downstream validation that can flag suspicious values for human review rather than assuming the model output is always correct.

Comparison table: full decision framework

Dimension	Tesseract / EasyOCR	PaddleOCR / PaddleOCR-VL	GPT-4o / Gemini Flash	dots.ocr (1.7B)	Donut (fine-tuned)
Cost/page (indicative)	~$0.00001	~$0.0002 self-hosted	~$0.008–0.01 API	~$0.0003 self-hosted	~$0.00005 self-hosted
Clean print accuracy	High (CER <1%)	High	Very high	High	High (on target form)
Tables / complex layout	Low	Medium	High	High	Very high (on target form)
Handwriting	Very low	Low-medium	High	Medium	High (on target form)
Direct JSON output	No	Partial	Yes	Yes	Yes
Multilingual	Good (100+ with packs)	Good	Very good	Very good (100+ lang)	Limited to training
Debuggability	High	High	Medium	Medium	Low
Generalization	High	High	Very high	High	None (fixed form only)
Infrastructure needed	CPU	GPU recommended	API (no infra)	GPU	GPU + training data

The volume thresholds where each option wins:

• Below ~5,000 pages/day: GPT-4o or Gemini Flash API. Spend the money, save the engineering.
• 5,000–100,000 pages/day: dots.ocr or PaddleOCR-VL-1.5 self-hosted for complex documents; classical OCR for clean print.
• Above 100,000 pages/day: classical OCR for clean print is non-negotiable on cost. For complex documents at this scale, a fine-tuned Donut-style model on your specific form types is the only economically sane path — but it requires training data and a permanent maintenance commitment.

{ "type": "token-cost", "title": "What 10K pages/day costs on the API path" }

The managed middle path

The framework above skips a category most production teams evaluate first: managed document-AI services — AWS Textract, Azure Document Intelligence, Google Document AI. They sit between classical OCR and frontier VLM APIs: per-page pricing instead of per-token, dedicated tables and forms endpoints, per-field confidence scores, and a compliance posture (HIPAA eligibility, data-residency controls) that a raw model API does not hand you. Indicative pricing as of mid-2026: plain text detection around $0.0015/page, table extraction around $0.015/page, key-value form extraction up to $0.05/page — cheaper than a frontier VLM for text, more expensive once you turn on the structured endpoints. They are a sensible default when your documents match what the prebuilt models were tuned for (invoices, receipts, IDs, standard forms) and you want confidence scores and an SLA instead of a Pydantic retry loop. They lose to VLMs on free-form layouts and handwriting-heavy corpora, and to self-hosted models on price at high volume.

One more lever shifts the break-even math: batch API pricing. Document extraction is the canonical latency-insensitive workload — nobody needs an invoice parsed in 800 ms — and the OpenAI, Anthropic, and Gemini batch endpoints all run at roughly a 50% discount over synchronous pricing (as of mid-2026). That halves the VLM path in the back-of-envelope: ~$0.004–0.005/page, $40–50/day at 10K pages/day, and it pushes the volume threshold where self-hosting starts to pay for itself correspondingly higher. If your pipeline is offline, price the batch tier, not the synchronous one.

The document ingestion pipeline picture

Document extraction does not live in isolation. The typical production pipeline for RAG over PDFs and mixed corpora includes extraction as one stage in a longer flow. Understanding where extraction sits helps clarify which failure modes matter most.

flowchart LR
    PDF[PDF upload] --> RENDER["Render to images\n150 or 300 DPI"]
    RENDER --> ROUTE{Document type\nclassifier}
    ROUTE -->|Clean print| CLASSIC[Classical OCR\nPaddleOCR]
    ROUTE -->|Complex / mixed| VLM_EXT["VLM extraction\nGemini Flash / dots.ocr"]
    CLASSIC --> VALIDATE[Schema validation\nPydantic]
    VLM_EXT --> VALIDATE
    VALIDATE -->|Pass| CHUNK[Chunk for RAG\nor store structured]
    VALIDATE -->|Fail| REVIEW[Human review\nqueue]
    CHUNK --> INDEX[Vector index\nor structured DB]

The document type classifier is often the highest-ROI investment in the pipeline. A cheap, fast classifier (even a rule-based one using filename, page count, and a low-cost VLM call on the first page) routes documents to the extraction path that is accurate and cheapest for that type. Without routing, you are either overpaying for VLM processing of clean print invoices or getting poor accuracy on complex documents that needed VLM treatment.

For the ingestion pipeline in detail — chunking strategies, embedding choice, and retrieval architecture for the downstream RAG system — the document ingestion pipeline article covers that territory. This article stops at the extraction output.

When the judgment call changes

A few factors that shift the decision in non-obvious ways:

Regulatory compliance logs. Classical OCR + rule-based parsing leaves a clear audit trail: exactly what text was extracted, exactly what rule mapped it to a field. VLMs are opaque — you know the model returned a value, but you cannot replay the reasoning. For healthcare documents, financial filings, or any domain where extraction decisions must be auditable, the debuggability of classical OCR is a genuine advantage, not just a comfort preference.

Language mix. Tesseract requires explicit language pack configuration per document; dots.ocr supports 100+ languages without switching. For a global document pipeline processing French, Arabic, Chinese, and Portuguese documents in the same batch, the multilingual advantage of VLMs and specialized open-weight models is substantial.

Error cost asymmetry. In some workflows, a missed extraction (null returned when a value exists) is far cheaper to handle than a hallucinated extraction (wrong value returned confidently). Null values trigger a human review queue; wrong values propagate silently. VLMs hallucinate; classical OCR returns garbage characters that are easier to detect as wrong. For high-stakes extraction where wrong answers are worse than no answers, prefer classical OCR on the fields that matter most, even if you use a VLM for everything else.

The models and rankings described here are current as of mid-2026. OCR Arena rankings, API pricing, and available open-weight models shift quarterly — the decision framework is stable, but the specific model choices inside each quadrant should be re-evaluated against current benchmarks on your own document corpus before any major volume commitment.

For the vision API side of this — comparing GPT-4o, Claude, and Gemini for general image understanding, not document-specific extraction — calling vision APIs in production covers the cost and accuracy patterns across the full image input space. And if the reason you are extracting these documents is to build a retrieval system, the structural output from this pipeline feeds directly into the multimodal RAG architecture.