# Structured Outputs and Constrained Decoding: Guaranteeing Parseable JSON > How strict mode and grammar-constrained decoding actually guarantee parseable JSON — token masking, FSMs, xGrammar, and the validation layer you still need. Canonical: https://ironclad.academy/ai-engineering/articles/structured-outputs-constrained-decoding | Difficulty: advanced | Topics: Structured Outputs, Tool Use, Inference, LLM Internals Covers: OpenAI, Anthropic, Hugging Face ## Key takeaways - Constrained decoding guarantees structural validity, not semantic correctness — a hallucinated order ID that passes the schema is still wrong. - xGrammar compiles JSON Schemas to pushdown automata and is the default constrained decoding backend in vLLM, SGLang, and TensorRT-LLM as of 2025. - OpenAI strict mode is incompatible with parallel_tool_calls — you must pick one or set parallel_tool_calls: false explicitly. - Grammar compilation is amortized; the real latency cost of constrained decoding is token overhead when top tokens are masked, typically 5–20% on complex schemas. - Validation-retry loops catch what constrained decoding misses — semantic errors — and are the standard fallback for hosted APIs and older models. - The CRANE paper (arXiv 2502.09061, February 2025) documents that constrained decoding can hurt reasoning quality; for chain-of-thought tasks, generate reasoning unconstrained then extract the final structured answer separately. > **In one line:** Constrained decoding enforces JSON Schema at generation time by masking invalid tokens at every decoding step — guaranteeing parseable output, but not semantically correct output, which still requires a separate validation layer. **The mechanism.** When an LLM generates text, it picks one token at a time from its vocabulary (32k–256k entries depending on the tokenizer). Constrained decoding adds a binary mask at each step: any vocabulary token whose full byte sequence would make the partial output invalid against the target schema is zeroed out before sampling. The schema is compiled once into a finite-state machine or pushdown automaton; the mask is computed by running the current partial output through that automaton to determine the legal next-character set. The result: the model physically cannot generate syntactically invalid JSON or schema-violating structure. **What it does not guarantee.** The schema says `order_id` is an integer. Constrained decoding ensures the model emits an integer. It cannot ensure that integer exists in your orders table. Semantic correctness — valid foreign keys, in-range values, business-rule consistency — is a separate problem that requires a separate validation layer. **Key points.** - xGrammar (pushdown automaton approach) is the default constrained decoding engine in vLLM, SGLang, TensorRT-LLM, and MLC-LLM as of 2025. - OpenAI Structured Outputs (`strict: true`) and Anthropic strict tool schemas apply the same mechanism server-side. - Grammar compilation is a one-time cost per schema, cached across requests. Per-token masking adds microseconds, not milliseconds. - The CRANE paper (arXiv 2502.09061, February 2025) documents quality degradation on reasoning tasks when top tokens are masked — the model picks lower-probability alternatives and can produce structurally correct but semantically wrong output. - Validation-retry loops (Pydantic/Zod + re-invoke with the error) are the production complement, not the alternative. ``` Back-of-the-envelope: constrained decoding overhead Schema compilation (xGrammar): ~1–10ms per unique schema, amortized to ~0ms with caching. Per-token masking: ~10–100µs per token (CPU operation alongside GPU decode). Token overhead on complex schemas: 5–20% more tokens when top tokens are frequently masked. Example: generating a 50-field JSON object at $3/M output tokens (illustrative, mid-2026 pricing): Without overhead: 200 output tokens × $3/M = $0.0006 per call With 15% token overhead: 230 output tokens × $3/M = $0.00069 per call → ~$0.00009 per call extra. At 1M calls/day: ~$90/day extra for the guarantee. ``` **The architecture in one diagram.** ```mermaid flowchart LR SCHEMA["JSON Schema"] -->|compile once| FSM["FSM / PDA\n(xGrammar)"] FSM -->|per token| MASK["Token mask\n(vocab × 1 binary vector)"] LOGITS["Model logits\n(raw, all vocab)"] --> APPLY["Apply mask\n(zero invalid tokens)"] MASK --> APPLY APPLY --> SAMPLE["Sample\n(temperature / top-p)"] SAMPLE -->|next token| PARTIAL["Partial output"] PARTIAL -->|advance automaton| FSM PARTIAL -->|complete| OUTPUT["Valid JSON output"] style FSM fill:#0e7490,color:#fff style MASK fill:#0e7490,color:#fff style OUTPUT fill:#15803d,color:#fff ``` **Key design decisions.** | Decision | Constrained decoding | Validation-retry loop | |---|---|---| | Guarantee | Structural (schema shape) | Structural + semantic (business rules) | | Latency | Adds 5–20% token overhead | Adds 1–3 full inference round-trips on failure | | Works on hosted APIs | Yes (if provider supports strict mode) | Yes, always | | Works on local models | Yes (xGrammar, Outlines, lm-format-enforcer) | Yes, always | | Handles semantic errors | No | Yes, with custom validators | | Schema complexity limit | High (any valid JSON Schema subset) | Unlimited (code) | **If you have 60 seconds, say this.** "Constrained decoding works by compiling a JSON Schema into a state machine that tracks which tokens are legal at each position. Before sampling, a binary mask zeroes out every vocabulary token that would violate the schema. xGrammar does this efficiently — compiling once, caching the grammar, paying microseconds per token. OpenAI's strict mode and Anthropic's strict tool schemas use the same idea server-side. But structural validity is not semantic validity: a perfectly valid JSON with a hallucinated ID is still wrong. So production systems layer constrained decoding with Pydantic validation and a retry loop that feeds the error back to the model. The other footgun is the CRANE finding: when top tokens are all masked, the model picks lower-probability alternatives, and reasoning quality can degrade. For chain-of-thought tasks, generate the reasoning unconstrained and extract the final answer in a second call." Your pipeline called the model, parsed the response, and crashed. Not on a hallucination — on a `json.JSONDecodeError`. The model decided mid-JSON that a string value containing a comma needed no closing quote. You added a try/except and a retry. The retry also failed, different token, different parse error. You shipped with a fallback to `None` and called it a day. This is the story of every team that used JSON mode and assumed it meant "parseable JSON." It doesn't. JSON mode tells the model to try to emit JSON. Structured Outputs with constrained decoding tells the model's sampler that emitting invalid JSON is physically impossible. This article is about how that guarantee works mechanically, what it costs, and — importantly — what it still doesn't protect you from. ## How the sampler sees a schema To understand constrained decoding, you need to understand what the model is actually doing at each token step. After the forward pass, the model produces a vector of logits — one float per vocabulary token (GPT-4o uses a ~200k-token vocabulary via `o200k_base`; Llama 3.x uses 128k). A softmax converts those to probabilities, temperature scales the distribution, and top-p or min-p filtering narrows it further. Then the sampler draws one token. **Constrained decoding inserts a binary mask between the raw logits and the sampling step.** Before sampling, every token whose byte sequence would make the partial output syntactically invalid against the target grammar is set to negative infinity (effectively zero probability). The model then samples from whatever is left. The key insight: the mask is determined not by the model, but by an automaton running in parallel. The automaton tracks the exact parse state of the partial output — "we are inside a string value after a colon, following the key `order_id`" — and computes which byte sequences could legally appear next. Any vocabulary token whose full byte sequence would drive the automaton into an invalid state gets masked — checking the first byte alone would not be enough, because a token starting legally can still continue illegally (in an enum state expecting `"shipped"`, a token spelling `sx` must be rejected). Making this full-token check fast is xGrammar's core contribution: it precomputes the mask for most tokens per automaton state and validates the small set of context-dependent tokens at runtime. ```mermaid sequenceDiagram participant M as Model forward pass participant A as Automaton (xGrammar) participant S as Sampler Note over A: Compiled from JSON Schema once M->>S: logits [0.3, 0.1, 0.05, ...] A->>S: mask [1, 0, 1, 0, 1, ...] (binary) S->>S: logits[mask==0] = -inf S->>S: sample from remaining S-->>A: chosen token (advance automaton state) S-->>M: token appended to context ``` The automaton is stateful: after each token, it advances to a new parse state, ready to compute the next mask. The model and automaton run in lockstep. ## What automata actually get compiled JSON Schemas are compiled to **finite-state machines (FSMs)** for simple schemas and **pushdown automata (PDAs)** for schemas with recursive or context-free structure. The distinction matters because JSON is not a regular language — arrays and objects can be nested arbitrarily — so an FSM alone cannot track unbounded nesting depth. PDAs handle this with an explicit stack. [xGrammar](https://arxiv.org/html/2601.04426) (the library that became the default backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM by 2025) compiles grammars in a format called EBNF (Extended Backus-Naur Form) to PDAs. A JSON Schema like: ```json { "type": "object", "properties": { "order_id": { "type": "integer" }, "status": { "enum": ["pending", "shipped", "delivered"] }, "items": { "type": "array", "items": { "type": "string" } } }, "required": ["order_id", "status", "items"], "additionalProperties": false } ``` compiles to an automaton that knows, at every position in the output: - whether we are currently inside a key or a value - what the legal keys are (only `order_id`, `status`, `items`) - what types are legal for the current key - for `status`, what the three exact legal string values are — and nothing else - for `items`, that the value must be an array `[` and each element must be a string Grammar compilation is a one-time cost per unique schema. xGrammar caches compiled grammars keyed by schema hash, so a schema used for every request in your app pays the compilation cost once (typically 1–10ms for a moderately complex schema) and then zero per-request. ## The hosted-API story: OpenAI strict mode OpenAI shipped Structured Outputs with `strict: true` in August 2024, stabilized through 2025. The mechanism is server-side — you pass the schema in the API call, and the serving infrastructure applies constrained decoding without you managing xGrammar yourself. ```python from openai import OpenAI from pydantic import BaseModel client = OpenAI() class OrderExtraction(BaseModel): order_id: int status: str # will be enforced by schema items: list[str] response = client.beta.chat.completions.parse( model="gpt-4o-2024-11-20", messages=[ {"role": "system", "content": "Extract order details from the text."}, {"role": "user", "content": "Order #4821 for ['widget', 'gadget'] is now shipped."}, ], response_format=OrderExtraction, ) order = response.choices[0].message.parsed # typed OrderExtraction — but None on refusal, see below ``` The `parse` helper on `beta.chat.completions` handles the schema conversion from Pydantic and gives you back a typed object. There is one structural check you must add regardless: ```python # ALWAYS check this before accessing .parsed if response.choices[0].message.refusal: raise ValueError(f"Model refused: {response.choices[0].message.refusal}") order = response.choices[0].message.parsed ``` OpenAI Structured Outputs can return a **refusal** — a structured signal that the model declined to complete the request (typically content policy, not a bug). Code that skips this check and goes straight to `.parsed` will throw a `NoneType` attribute error on the first refusal, usually discovered in production. There is also a hard incompatibility to know: `strict: true` and `parallel_tool_calls: true` do not work together on OpenAI. If you need parallel tool calls, you must set `parallel_tool_calls: false` explicitly when using strict structured outputs. This is not flagged at schema upload time — it surfaces as a runtime error or silent fallback. For Anthropic, tool schemas support `strict: true` (as of 2025 beta) which enforces that tool call inputs always match the declared schema. The underlying mechanism is not publicly documented, but behavior is consistent with token-level enforcement on the tool input JSON specifically. ## Local models: Outlines, lm-format-enforcer, and xGrammar directly If you are running inference locally — vLLM, SGLang, a custom inference stack — you can apply constrained decoding at the framework level. Three libraries dominate: **[Outlines](https://github.com/dottxt-ai/outlines)** is the most mature Python library for grammar-constrained generation. It works with any HuggingFace model and compiles Pydantic models, JSON Schemas, regex patterns, or custom EBNF grammars: ```python import outlines from pydantic import BaseModel model = outlines.models.transformers("meta-llama/Llama-3.2-3B-Instruct") class Order(BaseModel): order_id: int status: str items: list[str] generator = outlines.generate.json(model, Order) result = generator("Extract the order: Order #4821 for widgets, shipped.") # result is a typed Order object — the model cannot deviate from this schema ``` **[lm-format-enforcer](https://github.com/noamgat/lm-format-enforcer)** takes a different approach: instead of compiling schemas upfront to FSMs, it applies character-level enforcement at each token using a token trie. It integrates with HuggingFace `generate()` directly and handles both JSON schemas and regex. It is lighter-weight to set up than Outlines but less expressive for complex recursive grammars. **xGrammar directly** is the right choice if you are using vLLM or SGLang, since it is already the default backend — in vLLM's offline API you pass a `GuidedDecodingParams` inside `SamplingParams` (the OpenAI-compatible server exposes the same thing as a `guided_json` field in `extra_body`): ```python from vllm import LLM, SamplingParams from vllm.sampling_params import GuidedDecodingParams schema = { "type": "object", "properties": { "order_id": {"type": "integer"}, "status": {"enum": ["pending", "shipped", "delivered"]}, }, "required": ["order_id", "status"], "additionalProperties": False, } llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct") params = SamplingParams(guided_decoding=GuidedDecodingParams(json=schema), temperature=0.0) outputs = llm.generate(["Extract: Order 4821 shipped"], params) # outputs[0].outputs[0].text is guaranteed to parse against schema ``` xGrammar-2 (January 2026, arXiv 2601.04426) added tag-triggered structure switching: the model can generate free text and then, when it emits a special trigger tag, the constrained decoding engages for only the structured portion. This is useful for chain-of-thought flows where you want unconstrained reasoning followed by a constrained final answer — more on that trade-off below. ```mermaid flowchart TD subgraph "xGrammar-2 tag-triggered mode" FREE["Free text generation\n(reasoning, explanation)"] -->|model emits trigger tag| SWITCH["Switch: engage grammar"] SWITCH --> CONSTRAINED["Constrained generation\n(JSON output only)"] CONSTRAINED --> DONE["Complete — valid JSON guaranteed"] end style SWITCH fill:#0e7490,color:#fff style CONSTRAINED fill:#15803d,color:#fff ``` ## The CRANE problem and token overhead The CRANE paper (arXiv 2502.09061, February 2025) put empirical numbers on what practitioners had noticed anecdotally: constrained decoding can degrade reasoning quality. The mechanism is straightforward. When the model's top-probability tokens are all masked (they would violate the schema), sampling is forced into lower-probability vocabulary. On reasoning-heavy tasks — math problems, multi-step logic — this matters because the model's internal reasoning is expressed through its highest-probability token choices. Forcing it to pick lower-probability tokens can break the chain. The practical implication: do not apply constrained decoding to the *reasoning* portion of a generation. Apply it only to the final structured output. xGrammar-2's tag-triggered mode is exactly this pattern. For hosted APIs, the equivalent is a two-call architecture: ```python # Call 1: unconstrained reasoning reasoning_response = client.chat.completions.create( model="gpt-4o", messages=[ {"role": "system", "content": "Think through the problem step by step, then output the answer."}, {"role": "user", "content": user_prompt}, ], ) reasoning_text = reasoning_response.choices[0].message.content # Call 2: constrained extraction from the reasoning extraction_response = client.beta.chat.completions.parse( model="gpt-4o", messages=[ {"role": "system", "content": "Extract the structured answer from this reasoning."}, {"role": "user", "content": reasoning_text}, ], response_format=MySchema, ) result = extraction_response.choices[0].message.parsed ``` This costs two inference calls, but the quality improvement on tasks with non-trivial reasoning is measurable enough to justify it for accuracy-critical paths. The other cost to track is **token overhead on complex schemas**. When the automaton is in a state where only a narrow set of characters are legal — say, inside an enum value mid-way through the string — the mask is very aggressive. Many tokens get zeroed. The model may need multiple tokens to produce a value it would have expressed in one token without the constraint. On schemas with many enum fields, deeply nested objects, or long required string patterns, this can add 5–20% to output token count. ``` Worked example: token overhead on an enum-heavy schema Schema: 12 fields, 6 of them enums with 3-8 values each. Model: Llama 3.1 8B (128k vocab) Without constrained decoding: Output: 180 tokens, $0.00054 at $3/M (illustrative) With constrained decoding (xGrammar): Output: ~207 tokens (+15%), $0.00062 at $3/M Per-call overhead: $0.00008 At 500k calls/day: ~$40/day extra for structural guarantees. Schema compilation: amortized to ~0 after first request. Per-token masking: ~50µs × 207 tokens = ~10ms CPU work per request (runs alongside GPU decode, not blocking it). ``` ## The validation layer you still need Constrained decoding solves structural validity. It does not solve any of these: - **Hallucinated foreign keys**: `"customer_id": 99274` validates as integer, but customer 99274 does not exist. - **Out-of-range values**: `"discount_percent": 127` is a valid number even if your business only allows 0–100. - **Logically inconsistent fields**: `"shipped_at": "2026-03-01"` with `"status": "pending"` is structurally valid. - **Truncated required information**: a `"summary"` field that contains `"N/A"` because the model chose the shortest valid string. The standard production pattern is **structured output + Pydantic validation + retry**: ```python from pydantic import BaseModel, field_validator, ValidationError import anthropic client = anthropic.Anthropic() class OrderUpdate(BaseModel): order_id: int status: str discount_percent: float @field_validator("discount_percent") @classmethod def validate_discount(cls, v: float) -> float: if not 0 <= v <= 100: raise ValueError(f"discount_percent must be 0-100, got {v}") return v @field_validator("status") @classmethod def validate_status(cls, v: str) -> str: valid = {"pending", "processing", "shipped", "delivered", "cancelled"} if v not in valid: raise ValueError(f"Unknown status '{v}', must be one of {valid}") return v def extract_with_retry(text: str, max_retries: int = 2) -> OrderUpdate: messages = [{"role": "user", "content": f"Extract order info: {text}"}] for attempt in range(max_retries + 1): response = client.messages.create( model="claude-sonnet-4-5", max_tokens=512, tools=[{ "name": "extract_order", "description": "Extract structured order information", "input_schema": OrderUpdate.model_json_schema(), }], tool_choice={"type": "tool", "name": "extract_order"}, messages=messages, ) if response.stop_reason == "max_tokens": raise RuntimeError("Truncated at max_tokens — the tool input is incomplete; raise the limit") tool_use = next(b for b in response.content if b.type == "tool_use") try: return OrderUpdate(**tool_use.input) except ValidationError as e: if attempt == max_retries: raise # Feed the error back to the model messages.extend([ {"role": "assistant", "content": response.content}, {"role": "user", "content": [ {"type": "tool_result", "tool_use_id": tool_use.id, "content": f"Validation failed: {e}. Please fix and retry."}, ]}, ]) raise RuntimeError("unreachable") ``` The retry message appends the validation error as a `tool_result` rather than a fresh user message. This keeps the conversation structure correct and gives the model the full context — what it output, and exactly what was wrong. Two retries handle most semantic errors; beyond that, you are likely dealing with a model that genuinely cannot extract the requested information from the input, and escalation or a manual fallback is more appropriate than burning more tokens. ## The strict mode schema requirements you will trip over OpenAI strict mode has schema requirements that are not always obvious: ```python # This schema will be REJECTED by OpenAI strict mode bad_schema = { "type": "object", "properties": { "order_id": {"type": "integer"}, "note": {"type": "string"}, # not in required }, "required": ["order_id"], # missing "note" # missing "additionalProperties": false } # This schema is ACCEPTED good_schema = { "type": "object", "properties": { "order_id": {"type": "integer"}, "note": {"type": ["string", "null"]}, # nullable, but still required }, "required": ["order_id", "note"], # ALL properties must be required "additionalProperties": False, # MANDATORY for strict mode } ``` The rules for strict mode: - All properties must appear in `required`. Optional fields must use `["type", "null"]` union. - `additionalProperties: false` is mandatory on all object schemas, including nested ones. - `$ref`, `anyOf`, `oneOf`, `allOf` are supported but must also satisfy the above rules for every subschema they reference. Missing `additionalProperties: false` is by far the most common schema rejection. It is rejected at schema upload time (not at generation time), which at least means you learn about it before your first production call. Anthropic's equivalent for tool schemas: `strict: true` on the tool definition enforces that inputs always match the schema exactly. As a 2025 beta feature, the documentation is thinner, but the practical behavior is consistent — use it when you need input validation guarantees on tool calls rather than just output parsing. ```mermaid flowchart TD Q1{"Hosted API?"} Q1 -->|OpenAI| Q2{"strict: true\nsupported?"} Q1 -->|Anthropic| Q3{"Tool schema\nwith strict: true?"} Q1 -->|Self-hosted| Q4{"Framework?"} Q2 -->|Yes| STRICT["Use strict: true\ncheck .refusal\nset parallel_tool_calls: false"] Q2 -->|No / older model| RETRY["Use JSON mode\n+ validation-retry loop"] Q3 -->|Yes| ANTHSTRICT["Use strict: true on tool\n+ Pydantic validation retry"] Q3 -->|No| RETRY Q4 -->|vLLM / SGLang| GUIDED["GuidedDecodingParams in SamplingParams\n(xGrammar backend)"] Q4 -->|HuggingFace| OUTLINES["Outlines or lm-format-enforcer"] Q4 -->|Other| RETRY style STRICT fill:#15803d,color:#fff style ANTHSTRICT fill:#15803d,color:#fff style GUIDED fill:#0e7490,color:#fff style OUTLINES fill:#0e7490,color:#fff style RETRY fill:#ffaa00,color:#0a0a0f ``` ## What breaks in production **Truncation at `max_tokens`.** Constrained decoding guarantees every emitted token keeps the output valid *so far* — it cannot force the closing braces to fit inside the token budget. If generation stops at the limit (`finish_reason == "length"` on OpenAI, `stop_reason == "max_tokens"` on Anthropic), you get a prefix of valid JSON that will not parse. This is the one way "guaranteed to parse" output doesn't. Check the stop reason before parsing, and size `max_tokens` to the schema's worst case: a 50-field object with 30-token string values needs ~1,800 tokens of headroom, not the 512 you copied from a tutorial. **Schema rejection at upload time.** With OpenAI strict mode, schema validation happens when you first register the schema (or on the first call with `response_format`). A missing `additionalProperties: false` on a nested object, found at 3 AM on a cold deploy, is more painful than finding it in a unit test. ```python # Catch schema issues early — run this in your test suite from openai import OpenAI client = OpenAI() def validate_schema_strict(schema: dict) -> None: """Probe OpenAI to validate the schema before going to production.""" try: client.beta.chat.completions.parse( model="gpt-4o-mini", messages=[{"role": "user", "content": "test"}], response_format={"type": "json_schema", "json_schema": {"name": "probe", "strict": True, "schema": schema}}, ) except Exception as e: raise ValueError(f"Schema rejected by OpenAI strict mode: {e}") from e ``` **The refusal you didn't handle.** Covered in the hosted-API section above: `message.parsed` is `None` when the model returns a `refusal`, and skipping the check earns you an `AttributeError` on the first content-policy hit. **Quality degradation on constrained reasoning, and token overhead.** Both covered in the CRANE section above — generate reasoning unconstrained and extract the answer in a second constrained pass, and profile token usage before enabling strict mode on high-volume endpoints (enum-heavy schemas run 15–25% heavier). **The `parallel_tool_calls` footgun.** The hard incompatibility is covered in the hosted-API section. The workaround worth knowing: a two-phase architecture — parallel non-strict calls in the [agentic loop](/ai-engineering/articles/tool-use-round-trip-agentic-loop), strict-mode extraction on each individual tool result. **Semantic errors that pass schema validation.** The most insidious failure mode. Constrained decoding makes it easy to assume the output is correct once it parses. It isn't. A model asked to extract an `invoice_total` from a table will emit a valid number — it just might be the subtotal, or the tax line, or a number from an adjacent row. Schema-conformant, semantically wrong. This is what Pydantic validators and domain-specific business-rule checks are for, and skipping them because "the schema enforces it" is a category error. ## When to use what **Use constrained decoding (strict mode / xGrammar) when:** - You need a structural guarantee: the output must parse, full stop, with zero parse-error retries. - You are building a high-volume extraction pipeline where retry latency compounds. - You are on a local model and can integrate xGrammar or Outlines directly into the serving stack. **Use a validation-retry loop without constrained decoding when:** - Your validation rules cannot be expressed in JSON Schema (business logic, foreign key checks, cross-field invariants). - You are on an older API endpoint or model that does not support strict mode. - You are applying chain-of-thought reasoning and want to avoid CRANE-style quality degradation in the reasoning portion. **In production, combine both.** Constrained decoding eliminates parse errors and schema mismatches. The retry loop catches semantic errors that the schema cannot express. Together they handle the full failure space. Separately, each leaves a gap. The right schema design is also part of the answer. A schema that expresses your domain constraints precisely — enums instead of free strings, integer ranges where your system enforces them, required vs nullable fields correctly set — narrows the semantic error space that constrained decoding cannot reach. For guidance on that, the [schema design article](/ai-engineering/articles/designing-tool-schemas-llm-agents) covers the full contract between schema and model behavior. For understanding where structured outputs fit into the broader tool-calling picture — the round-trip protocol, parallel calls, and how the model decides when to call a tool versus return text — see [JSON Mode vs Structured Outputs vs Tool Calling](/ai-engineering/articles/json-mode-vs-structured-outputs-vs-tool-calling) for the comparison, and [the tool-use round trip](/ai-engineering/articles/tool-use-round-trip-agentic-loop) for the loop mechanics. The constrained decoding guarantee is real and worth using. Just don't let "the schema enforces it" become a reason to skip the validation layer. The schema enforces the shape. The validation layer enforces the meaning. You need both. ## Frequently asked questions Q: What is the difference between JSON mode and Structured Outputs in OpenAI? A: JSON mode only guarantees syntactically valid JSON — the model can still omit required fields, invent extra keys, or produce wrong types. Structured Outputs with strict: true enforces the exact schema at generation time via constrained decoding: every token is selected from the subset of vocabulary tokens that keep the output valid against the schema. OpenAI now explicitly recommends against JSON mode when Structured Outputs is available. Q: How does constrained decoding work at the token level? A: At every decoding step the schema is compiled into a finite-state machine or pushdown automaton that tracks which characters are legal at the current position. A binary mask zeros out every vocabulary token whose full character sequence would take the output into an invalid state. The model then samples from the surviving (non-zero) logits only. xGrammar, which is the default backend in vLLM, SGLang, and TensorRT-LLM as of 2025, compiles JSON Schemas to pushdown automata and caches the compiled grammar across requests. Q: Does constrained decoding add latency? A: Grammar compilation is a one-time cost per unique schema, amortized across requests — xGrammar caches compiled grammars so subsequent requests with the same schema pay near-zero overhead. The per-token masking operation itself is fast (microseconds). The real cost is indirect: when the model's top tokens are all masked, it must pick from lower-probability alternatives, which can require more tokens to complete the same value, adding 5–20% token overhead on complex schemas in practice. Q: Does constrained decoding guarantee semantic correctness? A: No. Constrained decoding guarantees structural validity — the output parses and conforms to the schema — but it cannot prevent the model from hallucinating values that pass schema validation. An integer field will always be an integer, but it could be a hallucinated order ID that does not exist in your database. Semantic validation (range checks, foreign key lookups, business-rule assertions) is a separate layer you must build. Q: What is xGrammar and why does it matter? A: xGrammar is an open-source constrained decoding library that compiles JSON Schemas (and other context-free grammars) to pushdown automata and applies token masking efficiently on CPU alongside GPU decoding. It became the default constrained decoding backend in vLLM, SGLang, TensorRT-LLM, and MLC-LLM in late 2024 and 2025. xGrammar-2 (January 2026, arXiv 2601.04426) added tag-triggered structure switching for agentic workloads and cross-request grammar reuse. Q: When should I use a validation-retry loop instead of strict constrained decoding? A: Use a validation-retry loop when you are calling a hosted API (OpenAI, Anthropic) that gives you schema guarantees on the structure but no semantic guarantees, or when you need business-rule checks that cannot be expressed in JSON Schema. The pattern is: call the model, run Pydantic or Zod validation, and if it fails, append the error message as a user turn and re-invoke — up to 2–3 retries before raising. Retry loops are also the right fallback when you are on an older model or endpoint that does not support strict mode.