How Structured Output from LLMs Actually Works (And Why Your JSON Keeps Breaking)

JSON mode was never a guarantee. Here’s the mechanism behind constrained decoding, and the production gaps it doesn’t plug.

You called the model with response_format={"type": "json_object"}, parsed the response, and then the pipeline crashed on a Tuesday night because the model returned "confidence": "high" instead of a float. The JSON was valid. The schema was wrong. Your retry logic ran four times before the alert fired.

This is the distinction that breaks production systems: valid JSON and schema-compliant JSON are not the same thing. Most developers learn this in an incident, not before one.

The fix is not hard once you understand what the model is actually doing when it produces structured output. The mechanism is not magic and it is not prompting. It is constrained decoding: a form of token-level surgery that happens during inference, before any response reaches your application. Understanding it is the difference between bolting on a fallback and actually building a reliable pipeline.

What JSON Mode Actually Does (Not Much)

JSON mode (the type: "json_object" option in the OpenAI API) tells the model to produce syntactically valid JSON. That's the whole guarantee. The output will have balanced braces. Strings will have quotes. Numbers will be numbers.

It says nothing about which keys are present, what their types are, or whether the values make sense. You can ask for a { "price": 49.99, "currency": "MYR" } structure and receive { "amount": "49.99", "unit": "ringgit", "confidence": "very high" }. Valid JSON. Completely wrong schema.

JSON mode is useful for stopping the model from rambling in prose when you need structured data. It is not useful for guaranteeing what that structure looks like.

Here is what that looks like in practice.

Before — JSON mode, the version that breaks:

# Looks fine. Isn't.
response = client.chat.completions.create(
    model="gpt-5.4",
    messages=[
        {
            "role": "system",
            "content": (
                "Extract invoice details. Return JSON with fields: "
                "vendor (string), amount (float), currency (string), "
                "status (one of: paid, pending, overdue)."
            ),
        },
        {"role": "user", "content": invoice_text},
    ],
    response_format={"type": "json_object"},
)

import json
data = json.loads(response.choices[0].message.content)

# What you expected:
# {"vendor": "Acme Corp", "amount": 249.90, "currency": "MYR", "status": "paid"}
# What you might actually get:
# {"supplier": "Acme Corp", "total": "249.90", "currency": "ringgit", "status": "settled"}
# - "supplier" instead of "vendor"          → KeyError at runtime
# - "249.90" as a string instead of float   → TypeError in your calculation
# - "ringgit" instead of "MYR"             → passes through silently, breaks downstream
# - "settled" instead of "paid"            → your status check never matches

The model cooperated. The JSON is valid. Your pipeline still breaks.

After — Structured Outputs, the version that holds:

from openai import OpenAI
from pydantic import BaseModel
from typing import Literal

class Invoice(BaseModel):
    vendor: str
    amount: float
    currency: str
    status: Literal["paid", "pending", "overdue"]
client = OpenAI()
response = client.beta.chat.completions.parse(
    model="gpt-5.4",
    messages=[
        {"role": "system", "content": "Extract invoice details as structured data."},
        {"role": "user", "content": invoice_text},
    ],
    response_format=Invoice,
)
invoice = response.choices[0].message.parsed
# invoice.vendor   → str, always present
# invoice.amount   → float, always a number
# invoice.currency → str, always present
# invoice.status   → "paid" | "pending" | "overdue", nothing else possible

vendor will always be a string. amount will always be a float. status will always be one of those three values. The model cannot produce anything else. Not because you asked nicely in the prompt, but because the tokens for any other output are blocked at inference time.

OpenAI’s Structured Outputs, introduced in mid-2024, makes a harder promise: the model will always produce output that matches your JSON Schema exactly. Required fields will be present. Types will be correct. Enum values will be valid. When OpenAI introduced Structured Outputs in 2024, internal evals on complex schema adherence showed the original gpt-4o with Structured Outputs scoring 100%, compared to below 40% for older models using JSON mode. The GPT-5 series, current as of 2026, maintains that guarantee across gpt-5.4, gpt-5.4-mini, and gpt-5.4-nano.

That is not a small difference. It is the difference between a feature you can ship and one you cannot.

How Constrained Decoding Works

The mechanism behind Structured Outputs is constrained decoding. To understand it, you first need to understand what an LLM is doing at the moment it generates each token.

After each token, the model produces a probability distribution over its entire vocabulary, tens of thousands of possible next tokens. Normally, the runtime samples from this distribution (or takes the argmax, depending on temperature settings) and moves on. Constrained decoding intercepts that process. Before the next token is selected, the runtime masks out every token that would make the output invalid given where the generation is so far.

The question is: how does the runtime know which tokens are valid?

OpenAI’s implementation converts your JSON Schema into a context-free grammar (CFG). A CFG defines the set of all valid strings in a formal language: in this case, all JSON objects that satisfy your schema. At each generation step, the runtime uses the CFG to determine which tokens can legally follow what has been produced so far. Tokens that would violate the grammar get their logits set to negative infinity, which removes them from consideration before sampling.

This is why OpenAI chose CFGs over simpler approaches like finite state machines (FSMs). FSMs can represent regular languages. They handle flat JSON objects and fixed-depth nesting without trouble. But FSMs cannot represent recursive types. A schema with a tree node that contains an array of tree nodes is recursive, and an FSM-based approach cannot express which tokens are valid inside a deeply nested branch. CFGs handle this cleanly because they can define recursive production rules.

XGrammar, from CMU researchers, takes this further. It divides the vocabulary into context-independent tokens that can be pre-checked once for any position, and context-dependent tokens that require runtime evaluation. For JSON grammar tasks, XGrammar achieves token mask generation in under 40 microseconds, with up to a 100x speedup over Outlines (a popular grammar-constrained generation library) in CFG-guided tasks. Constrained generation adds nearly zero latency after the first call with a given schema, because the token masks are precomputed and cached.

Claude’s structured outputs use a similar approach: constrained sampling with compiled grammar artifacts. The Anthropic implementation is generally available for Claude Sonnet 4.5, Opus 4.5 and Haiku 4.5, and like OpenAI’s, it guarantees structural correctness.

The practical implication: when you use structured outputs from either provider, the model is not “trying harder” to follow your schema. It is physically incapable of producing a token that violates it. The constraint is in the inference runtime, not in the prompt.

The Three Approaches, Compared

There are three distinct points in the stack where you can enforce structured output, and they have very different guarantees.

Native structured outputs (OpenAI, Anthropic API)

• Guarantee: full structural correctness at inference time
• Works by: CFG or grammar-compiled constrained decoding in the provider’s runtime
• Best when: you are using a supported model and your schema fits the provider’s constraints
• Limitation: schema restrictions apply (more on this below)

Library-level constrained generation (Outlines, Guidance, XGrammar)

• Guarantee: full structural correctness at inference time, for any model you run locally
• Works by: compiling your schema into a grammar and masking logits before sampling
• Best when: you are self-hosting a model (Llama, Mistral, Qwen) and need guaranteed output
• Limitation: adds infrastructure complexity; requires direct access to the model’s logits

Prompted output + post-hoc validation (the old way)

• Guarantee: none at inference time; validation happens after the response arrives
• Works by: asking the model to produce JSON, then parsing and validating the string
• Best when: you are using a model or API that doesn’t support native structured outputs
• Limitation: you will parse failures, you will retry, and you will still get semantic nonsense

Most teams start with prompted output because it requires no configuration. Most teams graduate to native structured outputs when their retry rate becomes embarrassing. The library approach is less common but essential for self-hosted deployments.

Where Structured Output Still Breaks

Constrained decoding solves structural correctness. It does not solve everything.

The optional field problem. OpenAI’s strict mode requires all fields in a schema to be declared as required. Optional fields in Python (the Optional[str] pattern) translate to Union[str, None] in Pydantic, which becomes an anyOf keyword in JSON Schema. The OpenAI API does not handle anyOf cleanly in strict mode. The workaround is to declare all fields as required and use nullable types explicitly: {"type": ["string", "null"]} instead of Optional[str]. If you are using Pydantic models without adjustment, your schema will fail schema validation on the API side before the model even runs.

The semantic validation gap. Constrained decoding guarantees that price is a float. It does not guarantee that the float is positive, or that it falls within a valid range, or that the currency field is a real ISO 4217 code rather than the string "ringgit". A structured output response of {"price": -500.0, "currency": "XYZ"} is structurally valid and semantically garbage.

This is why schema compliance from the provider is not the end of the pipeline. Pydantic validators, field validators and model validators, are the layer that catches semantic failures. Every field that has a meaningful constraint (price > 0, status in ALLOWED_STATUSES, email that is actually an email) needs a Pydantic validator, not just a type annotation. The provider enforces shape. Your application enforces meaning.

The fabrication problem, restated. Structured output cannot stop a model from fabricating a value that fits the type. If you ask for a confidence: float field, you will get a float. You might get 0.97 for a response the model has no business being confident about. The constraint ensures the type; it cannot audit the model's internal calibration.

The refusal edge case. Models will occasionally refuse to fill certain fields: fields requesting PII, harmful content, or anything the model’s safety layer flags. When this happens inside a structured output call, the behavior varies by provider. Some return a partial response. Some return an error. Design schemas to treat high-risk fields as nullable, and treat a null in a non-nullable field as a signal to check your safety configuration, not as a parsing bug.

Building a Production Validation Stack

A reliable structured output pipeline has three layers. None of them are optional.

1. Provider-level constraint. Use native structured outputs (OpenAI, Anthropic) or library-level constrained generation (Outlines on self-hosted). This eliminates structural failures entirely. If you are still using JSON mode plus a prompt instruction, you are running without a seatbelt and counting on the model to cooperate.
2. Schema-level validation. After the response arrives, validate it against your Pydantic model with validators, not just type coercion. Any field with a business-logic constraint needs a validator that enforces it. Treat a Pydantic ValidationError as a model failure, not a parsing failure. Log it. Retry if retrying makes sense. Escalate if it does not.
3. Semantic post-processing. For high-stakes output, run a second check that is not schema-based at all. A model extracting invoice amounts from text should have its numeric outputs sanity-checked against the source document. A model classifying customer intent should have its confidence scores calibrated against a holdout set. Constrained decoding does not validate against ground truth. Your application does.

The earlier “After” example gives you structural correctness. Adding semantic validators gives you all three layers in one place:

Full stack — structured output + semantic validation:

from openai import OpenAI
from pydantic import BaseModel, field_validator
from typing import Literal

class Invoice(BaseModel):
    vendor: str
    amount: float
    currency: str
    status: Literal["paid", "pending", "overdue"]
    # Layer 2: semantic validation - the provider won't catch these
    @field_validator("amount")
    @classmethod
    def amount_must_be_positive(cls, v: float) -> float:
        if v <= 0:
            raise ValueError("invoice amount must be positive")
        return v
    @field_validator("currency")
    @classmethod
    def currency_must_be_iso(cls, v: str) -> str:
        VALID_ISO = {"USD", "EUR", "GBP", "MYR", "SGD"}  # extend as needed
        if v.upper() not in VALID_ISO:
            raise ValueError(f"unrecognised currency code: {v}")
        return v.upper()
client = OpenAI()

def extract_invoice(text: str) -> Invoice:
    response = client.beta.chat.completions.parse(
        model="gpt-5.4",
        messages=[
            {"role": "system", "content": "Extract invoice details as structured data."},
            {"role": "user", "content": text},
        ],
        response_format=Invoice,  # Layer 1: provider-level constraint
    )
    result = response.choices[0].message.parsed
    # Pydantic validators ran during parse() - layer 2 is already active
    return result

parse() converts your model to a JSON Schema, enforces it at inference time, then runs your Pydantic validators before returning. A structural failure is now impossible. A semantic failure raises ValidationError — which you log, retry if useful, and escalate if not. That is the response you want: fail loudly at the application layer, not silently in production data.

What This Means for How You Design Schemas

Knowing how constrained decoding works changes how you design your Pydantic models for LLM output.

Keep schemas flat when you can. Deep nesting is expensive to compute over and more likely to hit edge cases in recursive type handling. If you need hierarchical output, consider splitting into multiple sequential calls rather than one deeply nested schema.

Avoid union types where possible. Optional[str] works, but it requires schema transformation for OpenAI strict mode. str | None declared explicitly as a nullable field causes fewer surprises. The Instructor library, a thin wrapper around the OpenAI and Anthropic APIs, handles this transformation automatically, which is one reason it is worth the dependency for teams doing heavy LLM-to-Pydantic integration.

Make enum fields narrow. The constrained decoder enforces that the value is one of your enum members. That is valuable. Make the enum set small enough that the model has a real chance of choosing the right member. An enum with 40 options and ambiguous labels is asking the model to make a classification decision that your schema cannot help with.

The Part Nobody Documents

Here is the thing constrained decoding cannot fix: the input. Every guarantee in this piece is about the output format. Nothing here guarantees that the model understood the question correctly, applied the right reasoning, or extracted the right value from an ambiguous document.

Structured output makes integration reliable. It does not make the model accurate. Those are separate problems, and conflating them is how teams end up with perfectly structured responses containing confident wrong answers.

Your schema is a pipe, not a filter. Shape your pipe carefully, validate what comes through it, and never mistake the shape of the output for the correctness of what’s in it.