Structured Outputs and Constrained Decoding: Guaranteeing Parseable JSON

Your pipeline called the model, parsed the response, and crashed. Not on a hallucination — on a json.JSONDecodeError. The model decided mid-JSON that a string value containing a comma needed no closing quote. You added a try/except and a retry. The retry also failed, different token, different parse error. You shipped with a fallback to None and called it a day.

This is the story of every team that used JSON mode and assumed it meant "parseable JSON." It doesn't. JSON mode tells the model to try to emit JSON. Structured Outputs with constrained decoding tells the model's sampler that emitting invalid JSON is physically impossible.

This article is about how that guarantee works mechanically, what it costs, and — importantly — what it still doesn't protect you from.

How the sampler sees a schema

To understand constrained decoding, you need to understand what the model is actually doing at each token step. After the forward pass, the model produces a vector of logits — one float per vocabulary token (GPT-4o uses a ~200k-token vocabulary via o200k_base; Llama 3.x uses 128k). A softmax converts those to probabilities, temperature scales the distribution, and top-p or min-p filtering narrows it further. Then the sampler draws one token.

Constrained decoding inserts a binary mask between the raw logits and the sampling step. Before sampling, every token whose byte sequence would make the partial output syntactically invalid against the target grammar is set to negative infinity (effectively zero probability). The model then samples from whatever is left.

The key insight: the mask is determined not by the model, but by an automaton running in parallel. The automaton tracks the exact parse state of the partial output — "we are inside a string value after a colon, following the key order_id" — and computes which byte sequences could legally appear next. Any vocabulary token whose full byte sequence would drive the automaton into an invalid state gets masked — checking the first byte alone would not be enough, because a token starting legally can still continue illegally (in an enum state expecting "shipped", a token spelling sx must be rejected). Making this full-token check fast is xGrammar's core contribution: it precomputes the mask for most tokens per automaton state and validates the small set of context-dependent tokens at runtime.

sequenceDiagram
    participant M as Model forward pass
    participant A as Automaton (xGrammar)
    participant S as Sampler

    Note over A: Compiled from JSON Schema once
    M->>S: logits [0.3, 0.1, 0.05, ...]
    A->>S: mask [1, 0, 1, 0, 1, ...] (binary)
    S->>S: logits[mask==0] = -inf
    S->>S: sample from remaining
    S-->>A: chosen token (advance automaton state)
    S-->>M: token appended to context

The automaton is stateful: after each token, it advances to a new parse state, ready to compute the next mask. The model and automaton run in lockstep.

What automata actually get compiled

JSON Schemas are compiled to finite-state machines (FSMs) for simple schemas and pushdown automata (PDAs) for schemas with recursive or context-free structure. The distinction matters because JSON is not a regular language — arrays and objects can be nested arbitrarily — so an FSM alone cannot track unbounded nesting depth. PDAs handle this with an explicit stack.

xGrammar (the library that became the default backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM by 2025) compiles grammars in a format called EBNF (Extended Backus-Naur Form) to PDAs. A JSON Schema like:

{
  "type": "object",
  "properties": {
    "order_id": { "type": "integer" },
    "status": { "enum": ["pending", "shipped", "delivered"] },
    "items": {
      "type": "array",
      "items": { "type": "string" }
    }
  },
  "required": ["order_id", "status", "items"],
  "additionalProperties": false
}

compiles to an automaton that knows, at every position in the output:

• whether we are currently inside a key or a value
• what the legal keys are (only order_id, status, items)
• what types are legal for the current key
• for status, what the three exact legal string values are — and nothing else
• for items, that the value must be an array [ and each element must be a string

Grammar compilation is a one-time cost per unique schema. xGrammar caches compiled grammars keyed by schema hash, so a schema used for every request in your app pays the compilation cost once (typically 1–10ms for a moderately complex schema) and then zero per-request.

The hosted-API story: OpenAI strict mode

OpenAI shipped Structured Outputs with strict: true in August 2024, stabilized through 2025. The mechanism is server-side — you pass the schema in the API call, and the serving infrastructure applies constrained decoding without you managing xGrammar yourself.

from openai import OpenAI
from pydantic import BaseModel

client = OpenAI()

class OrderExtraction(BaseModel):
    order_id: int
    status: str  # will be enforced by schema
    items: list[str]

response = client.beta.chat.completions.parse(
    model="gpt-4o-2024-11-20",
    messages=[
        {"role": "system", "content": "Extract order details from the text."},
        {"role": "user", "content": "Order #4821 for ['widget', 'gadget'] is now shipped."},
    ],
    response_format=OrderExtraction,
)

order = response.choices[0].message.parsed  # typed OrderExtraction — but None on refusal, see below

The parse helper on beta.chat.completions handles the schema conversion from Pydantic and gives you back a typed object. There is one structural check you must add regardless:

# ALWAYS check this before accessing .parsed
if response.choices[0].message.refusal:
    raise ValueError(f"Model refused: {response.choices[0].message.refusal}")

order = response.choices[0].message.parsed

OpenAI Structured Outputs can return a refusal — a structured signal that the model declined to complete the request (typically content policy, not a bug). Code that skips this check and goes straight to .parsed will throw a NoneType attribute error on the first refusal, usually discovered in production.

There is also a hard incompatibility to know: strict: true and parallel_tool_calls: true do not work together on OpenAI. If you need parallel tool calls, you must set parallel_tool_calls: false explicitly when using strict structured outputs. This is not flagged at schema upload time — it surfaces as a runtime error or silent fallback.

For Anthropic, tool schemas support strict: true (as of 2025 beta) which enforces that tool call inputs always match the declared schema. The underlying mechanism is not publicly documented, but behavior is consistent with token-level enforcement on the tool input JSON specifically.

Local models: Outlines, lm-format-enforcer, and xGrammar directly

If you are running inference locally — vLLM, SGLang, a custom inference stack — you can apply constrained decoding at the framework level. Three libraries dominate:

Outlines is the most mature Python library for grammar-constrained generation. It works with any HuggingFace model and compiles Pydantic models, JSON Schemas, regex patterns, or custom EBNF grammars:

import outlines
from pydantic import BaseModel

model = outlines.models.transformers("meta-llama/Llama-3.2-3B-Instruct")

class Order(BaseModel):
    order_id: int
    status: str
    items: list[str]

generator = outlines.generate.json(model, Order)
result = generator("Extract the order: Order #4821 for widgets, shipped.")
# result is a typed Order object — the model cannot deviate from this schema

lm-format-enforcer takes a different approach: instead of compiling schemas upfront to FSMs, it applies character-level enforcement at each token using a token trie. It integrates with HuggingFace generate() directly and handles both JSON schemas and regex. It is lighter-weight to set up than Outlines but less expressive for complex recursive grammars.

xGrammar directly is the right choice if you are using vLLM or SGLang, since it is already the default backend — in vLLM's offline API you pass a GuidedDecodingParams inside SamplingParams (the OpenAI-compatible server exposes the same thing as a guided_json field in extra_body):

from vllm import LLM, SamplingParams
from vllm.sampling_params import GuidedDecodingParams

schema = {
    "type": "object",
    "properties": {
        "order_id": {"type": "integer"},
        "status": {"enum": ["pending", "shipped", "delivered"]},
    },
    "required": ["order_id", "status"],
    "additionalProperties": False,
}

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
params = SamplingParams(guided_decoding=GuidedDecodingParams(json=schema), temperature=0.0)
outputs = llm.generate(["Extract: Order 4821 shipped"], params)
# outputs[0].outputs[0].text is guaranteed to parse against schema

xGrammar-2 (January 2026, arXiv 2601.04426) added tag-triggered structure switching: the model can generate free text and then, when it emits a special trigger tag, the constrained decoding engages for only the structured portion. This is useful for chain-of-thought flows where you want unconstrained reasoning followed by a constrained final answer — more on that trade-off below.

flowchart TD
    subgraph "xGrammar-2 tag-triggered mode"
        FREE["Free text generation\n(reasoning, explanation)"] -->|model emits trigger tag| SWITCH["Switch: engage grammar"]
        SWITCH --> CONSTRAINED["Constrained generation\n(JSON output only)"]
        CONSTRAINED --> DONE["Complete — valid JSON guaranteed"]
    end
    style SWITCH fill:#0e7490,color:#fff
    style CONSTRAINED fill:#15803d,color:#fff

The CRANE problem and token overhead

The CRANE paper (arXiv 2502.09061, February 2025) put empirical numbers on what practitioners had noticed anecdotally: constrained decoding can degrade reasoning quality. The mechanism is straightforward. When the model's top-probability tokens are all masked (they would violate the schema), sampling is forced into lower-probability vocabulary. On reasoning-heavy tasks — math problems, multi-step logic — this matters because the model's internal reasoning is expressed through its highest-probability token choices. Forcing it to pick lower-probability tokens can break the chain.

The practical implication: do not apply constrained decoding to the reasoning portion of a generation. Apply it only to the final structured output. xGrammar-2's tag-triggered mode is exactly this pattern. For hosted APIs, the equivalent is a two-call architecture:

# Call 1: unconstrained reasoning
reasoning_response = client.chat.completions.create(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Think through the problem step by step, then output the answer."},
        {"role": "user", "content": user_prompt},
    ],
)
reasoning_text = reasoning_response.choices[0].message.content

# Call 2: constrained extraction from the reasoning
extraction_response = client.beta.chat.completions.parse(
    model="gpt-4o",
    messages=[
        {"role": "system", "content": "Extract the structured answer from this reasoning."},
        {"role": "user", "content": reasoning_text},
    ],
    response_format=MySchema,
)
result = extraction_response.choices[0].message.parsed

This costs two inference calls, but the quality improvement on tasks with non-trivial reasoning is measurable enough to justify it for accuracy-critical paths.

The other cost to track is token overhead on complex schemas. When the automaton is in a state where only a narrow set of characters are legal — say, inside an enum value mid-way through the string — the mask is very aggressive. Many tokens get zeroed. The model may need multiple tokens to produce a value it would have expressed in one token without the constraint. On schemas with many enum fields, deeply nested objects, or long required string patterns, this can add 5–20% to output token count.

Worked example: token overhead on an enum-heavy schema

Schema: 12 fields, 6 of them enums with 3-8 values each.
Model: Llama 3.1 8B (128k vocab)

Without constrained decoding:
  Output: 180 tokens, $0.00054 at $3/M (illustrative)

With constrained decoding (xGrammar):
  Output: ~207 tokens (+15%), $0.00062 at $3/M
  Per-call overhead: $0.00008

At 500k calls/day: ~$40/day extra for structural guarantees.
Schema compilation: amortized to ~0 after first request.
Per-token masking: ~50µs × 207 tokens = ~10ms CPU work per request (runs alongside GPU decode, not blocking it).

The validation layer you still need

Constrained decoding solves structural validity. It does not solve any of these:

• Hallucinated foreign keys: "customer_id": 99274 validates as integer, but customer 99274 does not exist.
• Out-of-range values: "discount_percent": 127 is a valid number even if your business only allows 0–100.
• Logically inconsistent fields: "shipped_at": "2026-03-01" with "status": "pending" is structurally valid.
• Truncated required information: a "summary" field that contains "N/A" because the model chose the shortest valid string.

The standard production pattern is structured output + Pydantic validation + retry:

from pydantic import BaseModel, field_validator, ValidationError
import anthropic

client = anthropic.Anthropic()

class OrderUpdate(BaseModel):
    order_id: int
    status: str
    discount_percent: float

    @field_validator("discount_percent")
    @classmethod
    def validate_discount(cls, v: float) -> float:
        if not 0 <= v <= 100:
            raise ValueError(f"discount_percent must be 0-100, got {v}")
        return v

    @field_validator("status")
    @classmethod
    def validate_status(cls, v: str) -> str:
        valid = {"pending", "processing", "shipped", "delivered", "cancelled"}
        if v not in valid:
            raise ValueError(f"Unknown status '{v}', must be one of {valid}")
        return v

def extract_with_retry(text: str, max_retries: int = 2) -> OrderUpdate:
    messages = [{"role": "user", "content": f"Extract order info: {text}"}]
    
    for attempt in range(max_retries + 1):
        response = client.messages.create(
            model="claude-sonnet-4-5",
            max_tokens=512,
            tools=[{
                "name": "extract_order",
                "description": "Extract structured order information",
                "input_schema": OrderUpdate.model_json_schema(),
            }],
            tool_choice={"type": "tool", "name": "extract_order"},
            messages=messages,
        )
        
        if response.stop_reason == "max_tokens":
            raise RuntimeError("Truncated at max_tokens — the tool input is incomplete; raise the limit")
        
        tool_use = next(b for b in response.content if b.type == "tool_use")
        
        try:
            return OrderUpdate(**tool_use.input)
        except ValidationError as e:
            if attempt == max_retries:
                raise
            # Feed the error back to the model
            messages.extend([
                {"role": "assistant", "content": response.content},
                {"role": "user", "content": [
                    {"type": "tool_result", "tool_use_id": tool_use.id,
                     "content": f"Validation failed: {e}. Please fix and retry."},
                ]},
            ])
    
    raise RuntimeError("unreachable")

The retry message appends the validation error as a tool_result rather than a fresh user message. This keeps the conversation structure correct and gives the model the full context — what it output, and exactly what was wrong. Two retries handle most semantic errors; beyond that, you are likely dealing with a model that genuinely cannot extract the requested information from the input, and escalation or a manual fallback is more appropriate than burning more tokens.

The strict mode schema requirements you will trip over

OpenAI strict mode has schema requirements that are not always obvious:

# This schema will be REJECTED by OpenAI strict mode
bad_schema = {
    "type": "object",
    "properties": {
        "order_id": {"type": "integer"},
        "note": {"type": "string"},  # not in required
    },
    "required": ["order_id"],  # missing "note"
    # missing "additionalProperties": false
}

# This schema is ACCEPTED
good_schema = {
    "type": "object",
    "properties": {
        "order_id": {"type": "integer"},
        "note": {"type": ["string", "null"]},  # nullable, but still required
    },
    "required": ["order_id", "note"],  # ALL properties must be required
    "additionalProperties": False,  # MANDATORY for strict mode
}

The rules for strict mode:

• All properties must appear in required. Optional fields must use ["type", "null"] union.
• additionalProperties: false is mandatory on all object schemas, including nested ones.
• $ref, anyOf, oneOf, allOf are supported but must also satisfy the above rules for every subschema they reference.

Missing additionalProperties: false is by far the most common schema rejection. It is rejected at schema upload time (not at generation time), which at least means you learn about it before your first production call.

Anthropic's equivalent for tool schemas: strict: true on the tool definition enforces that inputs always match the schema exactly. As a 2025 beta feature, the documentation is thinner, but the practical behavior is consistent — use it when you need input validation guarantees on tool calls rather than just output parsing.

flowchart TD
    Q1{"Hosted API?"}
    Q1 -->|OpenAI| Q2{"strict: true\nsupported?"}
    Q1 -->|Anthropic| Q3{"Tool schema\nwith strict: true?"}
    Q1 -->|Self-hosted| Q4{"Framework?"}

    Q2 -->|Yes| STRICT["Use strict: true\ncheck .refusal\nset parallel_tool_calls: false"]
    Q2 -->|No / older model| RETRY["Use JSON mode\n+ validation-retry loop"]

    Q3 -->|Yes| ANTHSTRICT["Use strict: true on tool\n+ Pydantic validation retry"]
    Q3 -->|No| RETRY

    Q4 -->|vLLM / SGLang| GUIDED["GuidedDecodingParams in SamplingParams\n(xGrammar backend)"]
    Q4 -->|HuggingFace| OUTLINES["Outlines or lm-format-enforcer"]
    Q4 -->|Other| RETRY

    style STRICT fill:#15803d,color:#fff
    style ANTHSTRICT fill:#15803d,color:#fff
    style GUIDED fill:#0e7490,color:#fff
    style OUTLINES fill:#0e7490,color:#fff
    style RETRY fill:#ffaa00,color:#0a0a0f

What breaks in production

Truncation at max_tokens. Constrained decoding guarantees every emitted token keeps the output valid so far — it cannot force the closing braces to fit inside the token budget. If generation stops at the limit (finish_reason == "length" on OpenAI, stop_reason == "max_tokens" on Anthropic), you get a prefix of valid JSON that will not parse. This is the one way "guaranteed to parse" output doesn't. Check the stop reason before parsing, and size max_tokens to the schema's worst case: a 50-field object with 30-token string values needs ~1,800 tokens of headroom, not the 512 you copied from a tutorial.

Schema rejection at upload time. With OpenAI strict mode, schema validation happens when you first register the schema (or on the first call with response_format). A missing additionalProperties: false on a nested object, found at 3 AM on a cold deploy, is more painful than finding it in a unit test.

# Catch schema issues early — run this in your test suite
from openai import OpenAI
client = OpenAI()

def validate_schema_strict(schema: dict) -> None:
    """Probe OpenAI to validate the schema before going to production."""
    try:
        client.beta.chat.completions.parse(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": "test"}],
            response_format={"type": "json_schema", "json_schema": {"name": "probe", "strict": True, "schema": schema}},
        )
    except Exception as e:
        raise ValueError(f"Schema rejected by OpenAI strict mode: {e}") from e

The refusal you didn't handle. Covered in the hosted-API section above: message.parsed is None when the model returns a refusal, and skipping the check earns you an AttributeError on the first content-policy hit.

Quality degradation on constrained reasoning, and token overhead. Both covered in the CRANE section above — generate reasoning unconstrained and extract the answer in a second constrained pass, and profile token usage before enabling strict mode on high-volume endpoints (enum-heavy schemas run 15–25% heavier).

The parallel_tool_calls footgun. The hard incompatibility is covered in the hosted-API section. The workaround worth knowing: a two-phase architecture — parallel non-strict calls in the agentic loop, strict-mode extraction on each individual tool result.

Semantic errors that pass schema validation. The most insidious failure mode. Constrained decoding makes it easy to assume the output is correct once it parses. It isn't. A model asked to extract an invoice_total from a table will emit a valid number — it just might be the subtotal, or the tax line, or a number from an adjacent row. Schema-conformant, semantically wrong. This is what Pydantic validators and domain-specific business-rule checks are for, and skipping them because "the schema enforces it" is a category error.

When to use what

Use constrained decoding (strict mode / xGrammar) when:

• You need a structural guarantee: the output must parse, full stop, with zero parse-error retries.
• You are building a high-volume extraction pipeline where retry latency compounds.
• You are on a local model and can integrate xGrammar or Outlines directly into the serving stack.

Use a validation-retry loop without constrained decoding when:

• Your validation rules cannot be expressed in JSON Schema (business logic, foreign key checks, cross-field invariants).
• You are on an older API endpoint or model that does not support strict mode.
• You are applying chain-of-thought reasoning and want to avoid CRANE-style quality degradation in the reasoning portion.

In production, combine both. Constrained decoding eliminates parse errors and schema mismatches. The retry loop catches semantic errors that the schema cannot express. Together they handle the full failure space. Separately, each leaves a gap.

The right schema design is also part of the answer. A schema that expresses your domain constraints precisely — enums instead of free strings, integer ranges where your system enforces them, required vs nullable fields correctly set — narrows the semantic error space that constrained decoding cannot reach. For guidance on that, the schema design article covers the full contract between schema and model behavior.

For understanding where structured outputs fit into the broader tool-calling picture — the round-trip protocol, parallel calls, and how the model decides when to call a tool versus return text — see JSON Mode vs Structured Outputs vs Tool Calling for the comparison, and the tool-use round trip for the loop mechanics.

The constrained decoding guarantee is real and worth using. Just don't let "the schema enforces it" become a reason to skip the validation layer. The schema enforces the shape. The validation layer enforces the meaning. You need both.