Calling Vision APIs in Production: GPT-4o, Claude, and Gemini Compared

A startup's document processing pipeline launched in early 2025 using GPT-4o with detail: "auto" on every invoice image — including their 4000×3000 scans. Three weeks later, the API bill hit $18,000 against a budget of roughly $500. Their back-of-the-envelope had assumed low-detail pricing: 85 tokens, a fortieth of a cent per image. In production, auto chose high-detail mode on the large scans — 765 tokens per page, nine times the tokens — and each invoice averaged four scanned pages. A ~36× miss, compounding quietly at 90K invoices a day. Nobody had read the encoding docs.

That's not an edge case. It's the most common first mistake engineers make when calling vision APIs at scale. The second most common is treating the three major providers as interchangeable — same image in, same quality out, just different pricing. They are not interchangeable. The differences in accuracy profile, failure modes, and output structure are significant enough to matter in production, and they favor different models for different task types.

This article is about knowing those differences before they show up in your bill or your bug tracker.

How images become tokens

Understanding the cost model requires understanding how each provider turns pixels into the tokens you pay for.

GPT-4o uses a tiling system. Images arrive in one of two modes you specify: low detail and high detail (or auto, which picks based on image size). In low-detail mode, any image is resized to fit within 512×512 and costs a flat 85 tokens. In high-detail mode, the image is first scaled to fit within 2048×2048, then — the step everyone misses — its shortest side is downscaled to 768px, and only then is it divided into 512×512 tiles, each costing 170 tokens, plus a base charge of 85 tokens. That second downscale caps the cost. A 1024×1024 image becomes 768×768 and produces 4 tiles: 4 × 170 + 85 = 765 tokens. A 2048×2048 image also lands at 768×768 — the same 765 tokens. A 4000×3000 image — a typical smartphone photo — resizes to 2048×1536, then to 1024×768, then tiles 2×2: 4 × 170 + 85 = 765 tokens again. The worst case is an extreme aspect ratio: 768×2048 tiles into 2×4 = 8 tiles → 8 × 170 + 85 = 1,445 tokens. But the exact resize-then-tile behavior has edge cases: always verify actual token usage by inspecting the usage field in the API response on your real image distribution before estimating cost. OpenAI publishes the tiling formula, but the interaction between resize-first and tile-count is non-obvious.

Claude charges image inputs based on pixel count: Anthropic documents tokens ≈ (width × height) / 750, with images downscaled to at most 1,568px on the long edge. A 750×750 image is 750 tokens; the downscale cap means a typical image tops out around 1,600 tokens ($0.005 at ~$3/M input) no matter how large the original. The practical effect is similar to GPT-4o high-detail — both land in the high-hundreds-to-~1,600-token range for a photo — so the per-image cost difference between the two is less about the formula and more about the token price tier you're on.

Gemini 2.0 Flash charges 258 tokens for any image that fits within 384×384; larger images are tiled into 768×768 crops at 258 tokens each. (The flat-258-regardless-of-size behavior was Gemini 1.5 — 2.0 tiles.) A 2048×2048 image is ~9 crops ≈ 2,322 tokens, but at $0.10 per million input tokens that's still only ~$0.00023 per image — roughly 10–25× cheaper than GPT-4o in high-detail mode for a typical photograph, and the gap comes mostly from the per-token price (30× lower), not the token count. For bulk image processing pipelines where accuracy precision can be slightly lower, this cost gap is decisive.

flowchart TD
    IMG[Your image] --> CHECK{Detail level?}
    CHECK -->|low / auto small| FLAT["GPT-4o: flat 85 tokens\n≈ $0.00026"]
    CHECK -->|high / auto large| TILE["GPT-4o: shortest side → 768px,\nthen 512×512 tiles, 170 tokens/tile + 85 base\n≈ $0.0023–0.0043 for typical photos"]
    IMG --> CLAUDE["Claude: (w×h)/750 tokens,\ncapped ~1,600 by 1,568px downscale\n≈ $0.002–0.005"]
    IMG --> GEMINI["Gemini 2.0 Flash: 258 tokens\nper 768×768 crop\n≈ $0.0001–0.0003 typical photo"]

    style FLAT fill:#15803d,color:#fff
    style TILE fill:#00e5ff,color:#0a0a0f
    style CLAUDE fill:#a855f7,color:#fff
    style GEMINI fill:#0e7490,color:#fff

The practical recommendation: default to low detail for GPT-4o unless your task requires reading fine print or distinguishing small objects. For document extraction from scanned pages, you do need high detail, but then the cost math starts favoring Gemini or self-hosted models at scale. See the document AI comparison for that specific decision.

Before any of that: resize client-side. Every provider is going to downscale your image anyway — GPT-4o to 768px on the shortest side, Claude to 1,568px on the long edge, Gemini into 768×768 crops — so shipping the original 4000×3000 scan buys you nothing but upload latency. For Claude and Gemini, pre-resizing to those targets directly cuts token cost (fewer pixels, fewer crops). For GPT-4o, the post-downscale size sets the token count, but resizing to 512×512 yourself lets you choose low-detail mode deliberately instead of letting auto decide, and resizing to 768 on the shortest side makes the tile count predictable. One line of Pillow before upload is the fix for the $18,000 hook at the top of this article — cheaper and more controllable than any prompt or mode change.

Calling the APIs: what actually differs

The message structure is almost identical across providers, which is deceptive — it suggests the models are interchangeable.

GPT-4o (via OpenAI SDK):

from openai import OpenAI
import base64

client = OpenAI()

# Base64 image — or use a URL directly
with open("invoice.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/png;base64,{image_data}",
                        "detail": "high",   # or "low" or "auto"
                    },
                },
                {
                    "type": "text",
                    "text": "Extract the invoice number, date, and total amount.",
                },
            ],
        }
    ],
    # Structured output: enforce JSON schema
    response_format={
        "type": "json_schema",
        "json_schema": {
            "name": "invoice",
            "strict": True,
            "schema": {
                "type": "object",
                "properties": {
                    "invoice_number": {"type": ["string", "null"]},
                    "date": {"type": ["string", "null"]},
                    "total_amount": {"type": ["number", "null"]},
                },
                "required": ["invoice_number", "date", "total_amount"],
                "additionalProperties": False,
            },
        },
    },
)

# Always check actual token usage
print(response.usage)  # input_tokens includes the image tile count
data = response.choices[0].message.content

Claude (via Anthropic SDK):

import anthropic
import base64

client = anthropic.Anthropic()

with open("invoice.png", "rb") as f:
    image_data = base64.b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": (
                        "Extract the invoice number, date, and total amount. "
                        "Output JSON with keys: invoice_number (string or null), "
                        "date (string or null), total_amount (number or null). "
                        "If a field is not visible in the image, output null — "
                        "do not infer or guess."
                    ),
                },
            ],
        }
    ],
)

# Claude supports tool use for typed extraction, or text + json parsing
# For strict schemas, tool-use is more reliable than free-form JSON
print(message.usage)  # input_tokens includes image encoding

Gemini (via Google Generative AI SDK):

import google.generativeai as genai
import PIL.Image

genai.configure(api_key="YOUR_API_KEY")
model = genai.GenerativeModel("gemini-2.0-flash")

image = PIL.Image.open("invoice.png")

response = model.generate_content(
    [
        image,
        (
            "Extract the invoice number, date, and total amount. "
            "Respond with JSON only: "
            '{"invoice_number": ..., "date": ..., "total_amount": ...}'
        ),
    ],
    generation_config=genai.GenerationConfig(
        response_mime_type="application/json",
    ),
)

print(response.usage_metadata)  # total_token_count
data = response.text  # guaranteed JSON string with mime_type set

One ordering detail worth getting right: put images before text. Anthropic explicitly documents image-before-text as the best-practice ordering for Claude, and it is a sensible default for GPT-4o and Gemini too — text placed after the image can attend to the image tokens when forming its interpretation of the instruction. For multi-image prompts across all providers, put all images first, then the instruction at the end, so the model doesn't anchor on the text before seeing later images.

Structured output from vision: the reliability gap

Getting back valid JSON from a vision API is not guaranteed even when you ask for it explicitly. Here's what "structured output" actually means on each platform:

GPT-4o's response_format with "strict": True uses constrained decoding — the model's sampling is constrained to tokens that can extend a valid JSON document matching your schema. This means the JSON is syntactically valid and schema-conformant by construction. Missing required fields do not happen at the syntax level, though semantically null values can appear when the model can't extract a field. For vision tasks with complex schemas (10+ fields, nested objects), GPT-4o structured output is the most consistent choice available through an API as of mid-2026.

Claude does not offer schema-constrained decoding for text responses at the time of writing, but its tool-use API gives you typed extraction that behaves similarly. Define a tool with an input schema, ask the model to call it with the extracted data, and you get back a structured tool_use content block. The indirection is slightly verbose but the schema adherence is good and the error handling is cleaner than parsing free-form JSON.

# Claude tool-use for structured extraction
import anthropic, json

client = anthropic.Anthropic()

tools = [
    {
        "name": "extract_invoice",
        "description": "Extract structured data from the invoice image.",
        "input_schema": {
            "type": "object",
            "properties": {
                "invoice_number": {
                    "type": ["string", "null"],
                    "description": "Invoice or order number if visible",
                },
                "date": {
                    "type": ["string", "null"],
                    "description": "Invoice date in ISO 8601 format if parseable, else raw string",
                },
                "total_amount": {
                    "type": ["number", "null"],
                    "description": "Total amount due as a number, no currency symbol",
                },
            },
            "required": ["invoice_number", "date", "total_amount"],
        },
    }
]

message = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=tools,
    tool_choice={"type": "tool", "name": "extract_invoice"},
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {"type": "base64", "media_type": "image/png", "data": image_data},
                },
                {"type": "text", "text": "Extract the invoice data."},
            ],
        }
    ],
)

# Find the tool_use block
for block in message.content:
    if block.type == "tool_use":
        extracted = block.input  # already a dict, no json.loads needed
        break

Gemini's responseMimeType: "application/json" enforces valid JSON output but does not constrain to a specific schema. For simple flat schemas it works well; for deeply nested or polymorphic schemas, you get valid JSON that may not match your schema. The workaround is to include the schema in the prompt itself, which helps significantly but is not a guarantee.

The practical hierarchy for schema adherence: GPT-4o strict > Claude tool-use > Gemini JSON mode > all-three free-form JSON.

Always validate client-side regardless. Even with constrained decoding, nulls can appear where you expected values, or a number field may come back as a string in older client SDK versions. A one-liner Pydantic parse is cheap insurance:

from pydantic import BaseModel
from typing import Optional
import json

class Invoice(BaseModel):
    invoice_number: Optional[str]
    date: Optional[str]
    total_amount: Optional[float]

# After getting response text or tool input:
try:
    parsed = Invoice.model_validate(extracted)
except Exception as e:
    # Log, flag for human review, or retry
    print(f"Schema validation failed: {e}")

Where each model is actually better

The marketing claims are roughly accurate at a high level, but the specific task types where each model wins are worth naming precisely.

flowchart LR
    TASK[Extraction task] --> Q1{Dense table or\nmulti-column\nlayout?}
    Q1 -->|Yes| CLAUDE[Use Claude\nSonnet/Opus]
    Q1 -->|No| Q2{Fine-grained\ninstruction\nfollowing needed?}
    Q2 -->|Yes| GPT4O[Use GPT-4o]
    Q2 -->|No| Q3{High volume,\ncost critical, or\nlong video/context?}
    Q3 -->|Yes| GEMINI[Use Gemini 2.0 Flash]
    Q3 -->|No| GPT4O

    style CLAUDE fill:#a855f7,color:#fff
    style GPT4O fill:#15803d,color:#fff
    style GEMINI fill:#0e7490,color:#fff

Claude's strength: document layout. Claude consistently outperforms the other two on extracting structured data from dense, multi-column PDFs — financial statements, legal contracts, medical records, research papers. The likely reason is architectural: Anthropic describes Claude's vision processing as handling dynamic resolution tiling that preserves spatial relationships across column breaks better than fixed-size tiling. In practice, this shows up as fewer column-order transpositions (where text from column 2 gets merged with column 1 in the output) and more accurate table cell alignment.

GPT-4o's strength: instruction precision. When your extraction schema has specific rules — "if the date is ambiguous, prefer MM/DD/YYYY; if the amount appears multiple times, take the one labeled 'Total Due', not 'Subtotal'" — GPT-4o tends to follow them more reliably. This is consistent with its general text performance on instruction-heavy tasks. For receipts and forms with predictable structure but tricky business rules, GPT-4o high-detail is the standard recommendation.

Gemini's strength: throughput and context. Gemini 2.0 Flash is fast, cheap, and handles contexts that neither competitor can touch at comparable cost. Sending 50 images in a single request to extract a timeline across a photo album, or submitting a 30-minute video for scene analysis — these are use cases where Gemini's 1M token context and flat per-image pricing are structurally superior. For batch document processing pipelines that don't require table-extraction precision, Gemini 2.0 Flash is the cost-effective default.

A realistic accuracy comparison on standard document extraction tasks (illustrative, based on patterns from published evals as of early 2026, not a controlled benchmark):

Task	GPT-4o	Claude Sonnet	Gemini 2.0 Flash
Single-column invoice text extraction	~95% field accuracy	~94%	~91%
Multi-column financial table	~80%	~88%	~75%
Handwritten form fields	~78%	~76%	~72%
Printed receipt, clear scan	~96%	~95%	~93%
Diagram label extraction	~82%	~79%	~83%
Object count (≤5 objects)	~88%	~87%	~85%

These numbers drift as models are updated — treat them as directional, not contractual. Always benchmark on your specific document corpus before making a provider decision. The document AI article covers this evaluation approach in more depth.

Prompt patterns that actually move the needle

Getting better results from vision APIs is less about magic words and more about three specific structural changes.

Chain-of-thought before extraction. Ask the model to first describe what it sees in the image, then emit the structured output. This adds latency and tokens but measurably reduces key dropout — the failure mode where the model outputs null for a field that's clearly present in the image.

BAD (direct):
"Extract the invoice number, date, and total. Respond as JSON."

BETTER (chain-of-thought gate):
"First, describe what you see in this image in 2-3 sentences. 
Then extract: invoice number, date, total amount. 
Output format: first your description, then a JSON block with keys 
invoice_number, date, total_amount. If a field is not visible, output null — 
do not infer."

The description step forces the model to build a visual parse before committing to field values. When the image is ambiguous or partially visible, the description often surfaces the uncertainty in natural language, which you can use in a validation step.

Explicit null semantics. Every vision extraction prompt should say what to do when a field is absent. Models default to guessing or hallucinating plausible values when no instruction exists. "If a field is not visible in the image, output null" is not optional. Add "do not infer, do not guess, do not use values from similar documents" if your domain has recurring templates that might trigger memorized fills.

Negative constraints for high-stakes fields. For fields where a wrong value is worse than a null — tax IDs, medical dosages, contract amounts — add an explicit check: "Before outputting the total_amount, confirm it is labeled 'Total' or 'Total Due' in the image. If you see only a subtotal or partial amount, output null."

sequenceDiagram
    participant U as Your code
    participant M as Vision model
    participant V as Validator

    U->>M: Image + chain-of-thought prompt
    M-->>U: Description paragraph + JSON block
    U->>V: Parse JSON, run Pydantic schema check
    V-->>U: Valid / schema mismatch
    alt Schema mismatch
        U->>M: Retry with "Previous extraction failed validation: [error]. Try again."
        M-->>U: Corrected JSON
    end
    U-->>U: Store validated result

One retry with the validation error in context fixes the majority of schema mismatches without needing a more complex error-handling pipeline.

What breaks

These are the failure modes that are predictable enough to design around — not bugs that get fixed in the next model version, but structural limitations of how current VLMs process visual information.

Object counting. Every frontier VLM fails to reliably count objects in images when the count exceeds roughly 5–6. Ask GPT-4o how many cars are in a parking lot photo and you'll get a plausible number that is frequently wrong. The architectural reason is that the attention mechanism in transformer-based VLMs doesn't have an explicit counting register — it pattern-matches visual tokens to produce a number rather than actually enumerating. For counting tasks, use DETR, YOLO, or SAM-based detection pipelines and count the bounding boxes.

Spatial relationships. "Is the blue box to the left of the red box?" fails at a surprising rate — often 15–25% error on tasks that seem trivial to a human. "Which shelf does the product appear on?" is similarly unreliable. Spatial language grounding is an active research area and current production models have not solved it. Design your extraction schemas to avoid relying on spatial predicates; extract the objects and let your downstream logic handle spatial reasoning.

Negation queries. "Are there any people visible in this image?" has a false-negative rate of 10–20% in typical use — the model says "no" when a person is partially visible at the image edge, in shadow, or facing away. This is particularly dangerous for content moderation or safety-critical applications. If you need reliable absence detection, use dedicated classification models with explicit training on the negative class.

Low-confidence hallucination. When an image is blurry, poorly lit, or the text is small relative to image resolution, VLMs frequently hallucinate plausible-looking field values rather than outputting null. They do this more confidently than they should. The mitigation is dual-query consistency: send the same image with the same prompt twice, then diff the outputs. Fields where both responses agree are high-confidence; fields that differ are candidates for human review or null-out.

import json

def dual_query_extraction(client, image_data: str, prompt: str, schema_class) -> dict:
    """Run two independent extractions and flag disagreements."""
    results = []
    for _ in range(2):
        resp = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "user",
                    "content": [
                        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_data}", "detail": "high"}},
                        {"type": "text", "text": prompt},
                    ],
                }
            ],
            response_format={"type": "json_object"},
        )
        results.append(json.loads(resp.choices[0].message.content))

    a, b = results
    consensus = {}
    flagged = []
    # Union of keys: a key missing from one run IS a disagreement,
    # and json_object mode makes divergent key sets likely.
    for key in set(a) | set(b):
        if a.get(key) == b.get(key):
            consensus[key] = a[key]
        else:
            consensus[key] = None  # nullify disagreement
            flagged.append({"field": key, "run1": a.get(key), "run2": b.get(key)})

    schema_class.model_validate(consensus)  # raise early if consensus violates the schema
    return {"data": consensus, "flagged_for_review": flagged}

This doubles your API cost, so reserve it for high-value documents where a wrong extraction has real consequences.

Schema drift over model updates. OpenAI, Anthropic, and Google update their models on rolling schedules without deprecation warnings on minor versions. A schema that GPT-4o follows precisely today may start producing slightly different key names or value formats after a silent model update. Pin your extraction prompts to a specific model version (gpt-4o-2024-11-20 rather than gpt-4o) for production systems and run your extraction test suite against every version upgrade before promoting.

Sending multiple images

All three providers support multi-image inputs, but with different limits and performance characteristics.

GPT-4o accepts up to 10 images per API call. Claude accepts up to 20. Gemini's File API can handle thousands of images via file references, making it the only practical option for batch-in-a-single-request patterns.

For comparison tasks ("what changed between these two screenshots?"), all three perform reasonably well when the images are small (< 512px square in low-detail mode for GPT-4o). For "find the differences" tasks across high-resolution images, the token cost of high-detail mode on both images simultaneously can be prohibitive — budget accordingly.

For document workflows where you're processing pages of a multi-page PDF, the right pattern is not one API call per page. Convert pages to images, batch them (up to the provider's image limit per call), and send 5–10 pages at once with context about page ordering. This reduces API round-trips, reduces per-call overhead, and gives the model cross-page context for coherent extraction. See multimodal RAG for the full pipeline.

When to abandon APIs entirely

The unit economics of proprietary vision APIs break at high volume. At 100K images per day — a modest document processing business — you're looking at:

GPT-4o high-detail: 100K × $0.005 avg (incl. output tokens) = $500/day = $15,000/month
Claude Sonnet 4:    100K × $0.005 avg = $500/day = $15,000/month
Gemini 2.0 Flash:   100K × ~$0.0003 (large scans, ~12 crops) = $30/day = $900/month
Batch APIs:         ~50% off on all three → GPT-4o/Claude drop to ~$7,500/month

Self-hosted Qwen2.5-VL-72B on 8×A100 nodes:
  ~2,000 images/hour throughput per node
  100K images/day ≈ 50 node-hours/day (≈2 nodes running around the clock)
  Cloud GPU cost: ~$10–12/hour per node (A100 80GB)
  50 node-hours × $11 = $550/day = $16,500/month
  (With H100s and quantization: ~30% lower, ~$11,500/month)

Read that block honestly: self-hosting does not win on unit cost. At 100K images/day, self-hosted Qwen2.5-VL-72B lands at rough parity with GPT-4o pay-as-you-go, loses to the batch APIs, and is nowhere near Gemini Flash. And the batch tier is the underused lever here — all three providers offer roughly 50% discounts for asynchronous batch processing with results in hours, which fits most document pipelines fine. If your workload tolerates a delay, batch pricing should be your baseline before you compare anything against self-hosting. What self-hosting actually buys you is latency control, data residency, and customization via fine-tuning — reasons that can each justify the spend, but they are engineering reasons, not a line item on the bill. The architecture for those deployments is covered in the VLM internals article.

To run this math on your own workload, the estimator below helps — set input tokens to your per-image token count (tiles × 170 + 85 for GPT-4o high-detail; (w×h)/750 for Claude; 258 per crop for Gemini) multiplied by images per request, and requests per day to your daily image volume. It models text-token pricing, so it is the per-token cost that varies by model tier, not vision-specific tiling.

{ "type": "token-cost", "title": "Token cost explorer — set input tokens to your per-image count" }

The decision in practice

The model selection choice for vision API work comes down to three questions:

What's the primary task structure? Dense tables and complex PDF layouts → Claude. Precise instruction-following and OCR on clean documents → GPT-4o. High-volume batch processing, video, or cost-critical pipelines → Gemini 2.0 Flash.

What's your image volume? Below 10K images/month, any provider works and the cost difference is immaterial. Above 100K images/day, the gap between GPT-4o high-detail ($15K+/month pay-as-you-go, $7.5K on the batch API) and Gemini ($900/month) is a budget-level decision, not an engineering preference — and self-hosting ($11.5–16.5K/month for GPT-4o-level accuracy) enters the conversation for control reasons, not savings.

How strict is your schema? If you need guaranteed JSON schema adherence, GPT-4o structured output is the most consistent production option as of mid-2026. Claude tool-use is a close second with slightly more verbose error handling. Gemini JSON mode is sufficient for flat schemas.

One thing to get right regardless of provider: measure your actual token usage on real production images before committing to a pricing model. The estimates you calculate from documentation and the numbers you see in response.usage after calling the API on your actual image corpus will differ. Often significantly. The difference, projected to production volume, is what determines whether your product is economically viable.

For the architecture questions downstream of the API call — how to retrieve the right images for a user query, how to combine image and text context in a RAG pipeline — the multimodal RAG article picks up where this one ends.