JSON Mode vs Structured Outputs vs Tool Calling: What Actually Differs

Your extraction pipeline has been running fine for six weeks. Then a field goes missing in a response — confidence is just absent — and three downstream services throw KeyError in the same deploy window. No error was logged by the LLM. The JSON was syntactically valid. JSON mode did exactly what it promised: it returned valid JSON. It just didn't promise your schema.

That specific failure is the reason three distinct mechanisms exist. They look similar from the outside — you pass a schema, you get JSON back — but they enforce different contracts, fail in different ways, and carry different costs. Getting this wrong means either weaker guarantees than you assumed, or unnecessary complexity that adds latency and tokens for no benefit.

What JSON mode actually guarantees

JSON mode is a soft instruction. You tell the model to output JSON, and the model — being a next-token predictor — tries to comply. On strong frontier models with clean prompts, it succeeds almost all the time. "Almost" is the problem.

What JSON mode guarantees: the output will parse without a json.JSONDecodeError — provided generation actually finishes. Hit your max_tokens limit mid-object and even that fails (more on this in What breaks). That's it. Fields can be absent. Types can be wrong. Required nested objects can be replaced with null. The model can add fields you didn't ask for. None of this violates the JSON mode contract.

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="gpt-4o",
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": "Extract order data as JSON with fields: order_id (string), amount (float), currency (string)."
        },
        {
            "role": "user",
            "content": "Process order #A-2948 for $129.99"
        }
    ]
)

import json
data = json.loads(response.choices[0].message.content)
# data might be {"order_id": "A-2948", "amount": 129.99, "currency": "USD"}
# or it might be {"order_id": "A-2948", "total": 129.99}  ← "amount" missing
# or {"order_id": "A-2948", "amount": "129.99", "currency": "USD"}  ← wrong type

This is not hypothetical sloppiness — it happens under distributional shift (prompts the model hasn't seen variants of), under brevity pressure (the model decided currency was obvious from context and omitted it), or when the schema is complex enough that the model loses track of a nested requirement.

JSON mode is fine for exploratory work, for prototypes, and for cases where you control validation downstream and can handle malformed output gracefully. OpenAI itself says in its documentation: use Structured Outputs instead whenever possible.

Structured Outputs with strict mode

Structured Outputs changes the enforcement layer from "the model tries to follow instructions" to "the inference engine prevents invalid tokens from being generated."

At each decoding step, the server compiles your JSON schema into a grammar (a pushdown automaton or finite-state machine, depending on the implementation), then applies a binary mask over the vocabulary. Tokens that would produce output violating the grammar get zeroed out before sampling. The model samples from what's left. See structured outputs and constrained decoding for the full mechanism.

The result: a field cannot go missing because the grammar requires it. A string cannot appear where a number is required because " is masked out at that position. The output is schema-valid by construction, not by instruction.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class OrderExtraction(BaseModel):
    order_id: str
    amount: float
    currency: str

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",   # strict mode requires this or newer
    messages=[
        {
            "role": "system",
            "content": "Extract order data."
        },
        {
            "role": "user",
            "content": "Process order #A-2948 for $129.99"
        }
    ],
    response_format=OrderExtraction,
)

order = response.choices[0].message.parsed
# order.amount is guaranteed to be a float
# order.currency is guaranteed to be present
# order.order_id is guaranteed to be a string

The Python SDK's .parse() method handles the Pydantic-to-JSON-Schema conversion and sets strict: true automatically. Under the hood the schema sent to the API looks like:

{
  "type": "json_schema",
  "json_schema": {
    "name": "OrderExtraction",
    "strict": true,
    "schema": {
      "type": "object",
      "properties": {
        "order_id": {"type": "string"},
        "amount": {"type": "number"},
        "currency": {"type": "string"}
      },
      "required": ["order_id", "amount", "currency"],
      "additionalProperties": false
    }
  }
}

Those last two lines — required listing every field, additionalProperties: false — are not optional when using strict: true. They are required. Not because open schemas are impossible to constrain — JSON mode itself is a grammar over arbitrary valid JSON — but because with additionalProperties allowed, the grammar can no longer guarantee the output contains only the fields your code expects, which is the entire guarantee the feature exists to provide. So OpenAI only compiles closed schemas it can enforce completely, and rejects schemas that set strict: true without these constraints.

The refusal case

One failure mode that trips up people new to strict mode: the API can return a refusal instead of a parsed object. If the input triggers a content policy refusal, message.parsed is None and message.refusal contains the refusal string. Code that directly accesses message.parsed without checking will raise AttributeError or NullPointerException depending on your language.

if response.choices[0].message.refusal:
    # handle refusal — log it, return a fallback, escalate
    raise ValueError(f"Model refused: {response.choices[0].message.refusal}")

order = response.choices[0].message.parsed

This is not a JSON mode problem — JSON mode just returns text. It is specific to Structured Outputs because the strict API returns a typed response object with a clearly separable refusal path.

Tool calling: a different concept

Tool calling gets conflated with structured outputs because both return JSON. They are solving different problems.

Structured outputs: "format your response as this schema." Tool calling: "decide whether to invoke an external capability, and if so, with what arguments."

The distinction matters because tool calling changes the model's reasoning mode. When you provide tools, the model is not just formatting a response — it is deciding whether a function should be called at all, which function among possibly many, and what values to pass. The model returns a tool_use block (Anthropic) or tool_calls array (OpenAI) that your application code executes. The result comes back to the model as a tool_result message, and the model continues.

sequenceDiagram
    participant App as Your Application
    participant M as Model
    participant T as External Tool

    App->>M: user message + tool definitions
    M-->>App: stop_reason: tool_use<br/>tool_use {name: "get_order", input: {id: "A-2948"}}
    App->>T: execute get_order(id="A-2948")
    T-->>App: {status: "shipped", eta: "2026-03-25"}
    App->>M: tool_result {content: {...}}
    M-->>App: stop_reason: end_turn<br/>final answer text

The model sees the tool definitions as part of its reasoning context. The descriptions and parameter names are what it uses to decide when to call a tool and how to populate it — they function as a prompt, not just API documentation. This is covered in depth in designing tool schemas that models actually understand and the tool-use round trip.

For this article, the key point is: if you are trying to extract structured data from text that does not require any external call, tool calling is the wrong mechanism. You would be abusing a function-calling interface to get JSON formatting, which adds schema-in-the-tool-description overhead and forces the model into an action-selection mode when you just want a formatted answer. Use structured outputs.

Use tool calling when:

• The operation has a side effect (write to a database, call an API, send a message).
• The model needs to retrieve information it does not have in context.
• The model must choose between multiple possible actions.

Use structured outputs when:

• You want the model's answer in a specific JSON shape.
• You are extracting entities, classifying content, generating a report.
• No external system needs to be invoked.

{ "type": "agent-loop", "scenario": "weather-trip", "title": "Tool-use round trip: model decides, app executes, model continues" }

The parallel calls footgun

Both OpenAI and Anthropic support parallel tool calls: the model can return multiple tool_use blocks in one response. You execute them concurrently, collect all results, and send them back in a single message. This is good for performance — parallel independent queries run in wall-clock time equal to the slowest one rather than their sum.

On OpenAI, strict: true on tool schemas and parallel_tool_calls: true are mutually incompatible. The combination either errors or silently falls back. You must set:

response = client.chat.completions.create(
    model="gpt-4o",
    tools=[...],
    tool_choice="auto",
    parallel_tool_calls=False,   # required when strict: true on any tool
    messages=[...]
)

Missing this flag is one of the most common production bugs when migrating from JSON mode to strict structured outputs. It does not always error visibly — depending on the model version and the exact tool schema, it may silently disable strict enforcement instead of rejecting the request.

Anthropic's behavior differs. Parallel tool calls work fine with strict: true on client tools. The constraint only appears for server tools running in the same parallel group, which require explicit handling of turn boundaries. Check the Anthropic docs for the current server-tool rules, as the behavior has evolved through 2025.

Token overhead you are paying regardless

Every mechanism carries token costs beyond the content itself. This is not a reason to avoid them — it is a reason to be deliberate.

Token overhead comparison (illustrative, mid-2026 models):

JSON mode:
  • System prompt: your instructions only
  • Schema: must be in the prompt as natural language or examples
  • Per-request overhead: low (~100–300 tokens for schema description)

Structured Outputs:
  • Schema is injected by the API, not in your prompt
  • Per-request overhead: similar to JSON mode; schema compilation is server-side
  • No extra user-visible tokens; the schema is metadata

Tool calling (Anthropic):
  • Base tool-use system prompt: 290–804 tokens per request (varies by model + tool_choice)
  • Each tool definition: ~50–200 tokens depending on description length
  • 10 tools at 100 tokens each = 1,000 tokens per request
  • At $3/M input tokens and 50k req/day:
    1,000 tok × 50,000 req × $3/1M = $150/day in tool definitions alone

Mitigation: pass only the tools the current step needs, not your full tool catalog.
Anthropic's Tool Search Tool (2025) solves this for large catalogs: Claude loads tools on demand.

This math is why trimming your tool list matters at scale, and why Anthropic's Programmatic Tool Calling (launched November 2025 under advanced-tool-use-2025-11-20) reduces costs: Claude writes Python to orchestrate tool calls itself, so intermediate tool results stay in the code-execution environment instead of round-tripping through the model's context on every turn. On large workflows this cut billed input tokens by roughly 38% in Anthropic's benchmarks. (Loading definitions on demand is the Tool Search Tool's job, not PTC's.)

What breaks

Truncation voids the parse guarantee

Constrained decoding works token by token — it cannot conjure the closing braces if generation stops early. When the response hits your max_tokens limit, OpenAI returns finish_reason: "length" (Anthropic: stop_reason: "max_tokens") and hands you JSON cut off mid-object. json.loads throws, strict mode or not. OpenAI's own docs tell you to check finish_reason before parsing, and this is a genuinely common production bug: the schema that fit comfortably in tests grows a 40-item array in production and blows past the limit.

choice = response.choices[0]
if choice.finish_reason == "length":
    raise ValueError("Response truncated at max_tokens — do not parse")
data = json.loads(choice.message.content)

Size max_tokens for your worst-case payload, not your median one, and treat a length finish as a hard error rather than attempting to repair the fragment.

Schema conformance is not semantic correctness

This is the subtler failure and the one that survives strict mode. A perfectly schema-valid JSON object can contain a hallucinated order ID that does not exist in your database, a confidence value of 1.7 when your range is 0.0–1.0, or a currency of "Euros" when you specified ISO 4217 codes.

Constrained decoding enforces structural rules. It has no mechanism to enforce business logic. You need a separate validation layer:

from pydantic import BaseModel, field_validator

class OrderExtraction(BaseModel):
    order_id: str
    amount: float
    currency: str

    @field_validator("currency")
    @classmethod
    def must_be_iso_4217(cls, v):
        valid = {"USD", "EUR", "GBP", "JPY", "CAD", "AUD"}
        if v not in valid:
            raise ValueError(f"currency must be ISO 4217, got {v!r}")
        return v

    @field_validator("amount")
    @classmethod
    def must_be_positive(cls, v):
        if v <= 0:
            raise ValueError(f"amount must be positive, got {v}")
        return v

When Pydantic raises ValidationError, re-invoke the model with the error message appended as a user turn. This typically corrects semantic errors in one retry. Do not re-prompt from scratch — the error context is what helps.

Hallucinated tool names

When the model invents a tool that does not exist in your definitions, it returns a tool_use block with a name your application has no handler for. This is a tool-calling-specific failure that structured outputs do not have.

Guard against it with tool_choice: {"type": "tool", "name": "..."} when you know exactly which tool should be called, or tool_choice: "required" combined with validating the returned name against your registry before executing.

Schema rejection at compile time

Schemas that are technically valid JSON Schema but are not compilable to a finite automaton will be rejected by strict mode at request time. Common causes:

• Using oneOf or anyOf at the top level without a discriminator field.
• Recursive schemas (a Comment type that contains an array of Comment children).
• additionalProperties: true or absent while strict: true is set.

The error messages from OpenAI are reasonably descriptive. Test your schemas before deploying — compile them against the API once in your test suite and fail fast.

flowchart LR
    S["JSON Schema"] --> C["Grammar compiler\n(server-side, one-time)"]
    C -->|success| G["Compiled automaton\n(cached, reused)"]
    C -->|failure| E["Schema rejection error\nat request time"]
    G --> D["Inference: mask invalid\ntokens at each step"]
    D --> O["Schema-valid output\n(structural guarantee only)"]
    style E fill:#ff2e88,color:#111
    style O fill:#15803d,color:#fff

Context rot in tool loops

Tool calling introduces a loop: model call → tool execution → model call. Each turn appends to the context. After five or ten turns, the context contains the full history of every tool call and result, which can push you toward the context limit or shift the model's attention toward earlier turns and away from the current task.

The tool-use round trip article covers loop management in detail. The short version: cap iterations explicitly, summarize intermediate results when context exceeds a threshold, and never assume an agentic loop will converge on its own.

A concrete worked example: data extraction vs action invocation

Suppose you are building an order management system. Two tasks:

Task A: Parse an email and extract the order details. Use structured outputs. No external system needs to be called. The model reads the email and formats the answer.

class OrderDetails(BaseModel):
    order_id: str
    items: list[str]
    total_amount: float
    requested_delivery_date: str  # ISO 8601

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-08-06",
    messages=[
        {"role": "user", "content": f"Extract order details from: {email_text}"}
    ],
    response_format=OrderDetails,
)

Task B: Process the order — check inventory, create the order record, send a confirmation. Use tool calling. Three tools: check_inventory, create_order, send_confirmation. Each triggers an actual operation with side effects.

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_inventory",
            "description": "Check if the requested items are in stock. Returns availability status per item.",
            "parameters": {
                "type": "object",
                "properties": {
                    "item_ids": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "List of item SKUs to check"
                    }
                },
                "required": ["item_ids"],
                "additionalProperties": False
            },
            "strict": True
        }
    }
    # ... create_order, send_confirmation omitted for brevity
]

Notice strict: True on the tool schema. This ensures the model's tool arguments always match the schema. Notice additionalProperties: False — required for strict. And for this scenario, parallel_tool_calls=False if you set strict=True on any tool.

The two tasks use different mechanisms for good reason. Task A needs a formatted answer; Task B needs sequenced action invocation with real-world side effects.

Anthropic-specific behavior

On Anthropic's API, the parallel pattern is tool_use content blocks in the assistant message, and tool_result blocks in the subsequent user message. The stop_reason of "tool_use" signals that the model wants tools executed before it continues.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=1024,
    tools=[
        {
            "name": "get_weather",
            "description": "Get current weather for a location.",
            "input_schema": {
                "type": "object",
                "properties": {
                    "location": {"type": "string", "description": "City name"}
                },
                "required": ["location"]
            }
        }
    ],
    messages=[{"role": "user", "content": "What's the weather in Tokyo?"}]
)

if response.stop_reason == "tool_use":
    tool_use_block = next(b for b in response.content if b.type == "tool_use")
    tool_result = call_weather_api(tool_use_block.input["location"])

    # Continue the conversation
    follow_up = client.messages.create(
        model="claude-sonnet-4-5",
        max_tokens=1024,
        tools=[...],  # same tools
        messages=[
            {"role": "user", "content": "What's the weather in Tokyo?"},
            {"role": "assistant", "content": response.content},
            {
                "role": "user",
                "content": [
                    {
                        "type": "tool_result",
                        "tool_use_id": tool_use_block.id,
                        "content": str(tool_result)
                    }
                ]
            }
        ]
    )

For structured data extraction on Anthropic without tool calling, the standard approach is to describe the JSON schema in the system prompt (prefilling the assistant turn with { helps) — or use tool calling as a formatting trick (pass a single tool named extract_data and force its invocation with tool_choice: {"type": "tool", "name": "extract_data"}). Anthropic does not have a first-party strict structured output API as of mid-2026 that's exactly equivalent to OpenAI's response_format: json_schema with strict: true, though their tool strict: true mode (2025 beta) gives schema enforcement for tool inputs.

sequenceDiagram
    participant App
    participant Claude

    App->>Claude: user message + tools (stop_reason pending)
    Claude-->>App: stop_reason=tool_use<br/>content: [tool_use {id, name, input}]
    Note over App: Execute tool synchronously or async
    App->>Claude: messages with tool_result<br/>{tool_use_id, content}
    Claude-->>App: stop_reason=end_turn<br/>content: [text block]

The MCP angle

The Model Context Protocol (spec version 2025-06-18) added outputSchema to tool definitions. When a server provides outputSchema, it declares that its tool's return value will conform to that schema. Clients should validate tool results against it.

This is structured outputs extended to tool results: not just tool inputs (which strict mode already covers), but tool outputs too. It shifts validation left into the protocol layer, so your application doesn't have to write custom validators for every tool response.

{
  "name": "get_order_status",
  "description": "Returns the current status and estimated delivery of an order.",
  "inputSchema": {
    "type": "object",
    "properties": {
      "order_id": {"type": "string"}
    },
    "required": ["order_id"]
  },
  "outputSchema": {
    "type": "object",
    "properties": {
      "status": {"type": "string", "enum": ["processing", "shipped", "delivered"]},
      "eta": {"type": "string", "format": "date"}
    },
    "required": ["status"],
    "additionalProperties": false
  }
}

If you are building MCP servers or consuming tools through MCP clients, this is the pattern to follow. It makes the contract machine-readable at both ends of the call.

The decision in practice

Here is the actual decision process used in production:

Do you need to call an external API, execute code, read from a database, or take any action with a side effect? Tool calling. Full stop.

Do you need the model's answer in a specific JSON structure? Structured Outputs with strict: true. Define a Pydantic model, use the .parse() API, add semantic validators on top.

Are you working with an older model that does not support strict structured outputs, or a self-hosted model without constrained decoding? JSON mode with mandatory downstream validation. Treat every response as untrusted until validated. See structured data extraction with LLMs for validation patterns.

Is your schema too complex to satisfy strict mode's constraints (recursive types, dynamic field sets, polymorphic unions without discriminators)? Consider splitting the extraction into multiple simpler calls, or use JSON mode with strict Pydantic validation on the output.

One last thing that does not fit neatly into a table: the model's behavior shifts based on which mechanism you use. Tool calling puts the model in an action-selection mode — it is asking itself "should I call something, and if so what?" Structured outputs put it in a formatting mode — "how do I express this answer in the required shape?" For extraction tasks, structured outputs typically give better semantic quality than forcing extraction through a tool call, because the model's reasoning is oriented toward the right goal. For agentic tasks, tool calling is correct because the model needs to reason about whether to act at all.

Use the mechanism that matches the model's task, not just the output format you want.