Getting an LLM to return JSON you can actually parse.

Every AI feature has the same unglamorous joint in the middle: the model returns text, and your code needs data. Ask nicely and you'll get parseable JSON almost every time — and "almost" is exactly the problem, because a parser that fails one call in fifty is a bug report every day. Here's the ladder of guarantees, from prompting to schema-constrained decoding, and the failures that survive even the strongest rung.

Level 0: "Respond only with JSON"

The starting point is a sentence in the prompt: return a JSON object with keys x, y, z, and nothing else. It mostly works, and it fails in ways that are boringly predictable. The model wraps the object in a markdown code fence. It adds a preamble — "Here's the JSON you requested:" — because it's been trained to be helpful. It emits a trailing comma, or single quotes, or comments its own output. And when the response hits your output-token cap, the JSON simply stops mid-string, which no amount of prompting will fix.

If you're stuck at this level — some endpoints and older models leave you no choice — the survival kit is small: strip code fences before parsing, slice from the first { to the last }, and treat a parse failure as a retry, not a crash. But this is a floor, not a strategy.

Level 1: JSON mode

Most providers offer a switch that constrains the sampler itself so the model can only produce syntactically valid JSON — no fences, no preamble, no trailing commas. This kills the entire class of "it's JSON wearing a costume" failures, and if all you need is some valid object, it's enough.

What it doesn't promise is shape. Valid JSON can still be missing the key your code reads, or spell it subTitle instead of subtitle, or return a number as a string, or nest the fields one level deeper than yesterday. Your parse succeeds and your feature breaks one line later — arguably worse, because the error now surfaces far from its cause.

Level 2: schema-constrained output

The current state of the art is to hand the provider a JSON Schema and let it constrain decoding against it: at each step, tokens that would violate the schema are masked out, so the model cannot produce output of the wrong shape. OpenAI calls this Structured Outputs (a schema with strict enabled), Gemini takes a responseSchema in the generation config, and Claude takes a json_schema format via output_config. Same idea everywhere: required keys are present, types are correct, enums stay inside their allowed values, and unknown keys are gone — every single call.

Two practical notes. Strict modes generally want closed schemas — every object marked as allowing no additional properties, with an explicit required list — so generate the schema from your types rather than writing it by hand. And the first request with a new schema pays a one-time compilation cost before responses get fast (Claude caches compiled schemas for 24 hours), so don't benchmark on call number one.

The tool-call trick

Before native schema support existed, the reliable workaround was to abuse function calling: declare a tool named save_result whose input schema is the output you want, then force the model to call it. Tool arguments were always emitted as structured JSON, so the "function call" was really a schema-shaped envelope for your answer. The trick still earns its keep when you're already inside a tool-use loop and want the extraction to ride the same rails — and on Claude, marking the tool definition strict now gives forced calls the same hard validation guarantee as native structured output.

What a schema still can't catch

Schema-constrained decoding moves you from "parse error" to "guaranteed shape," and it's worth reaching for by default. Three failure classes survive it, and they're the ones that actually bite in production:

Truncation. If generation hits your max-token limit mid-object, no schema can conjure the closing brace. Check the finish reason before you parse — a response that stopped for length is incomplete by definition, whatever mode you requested. This compounds with reasoning models, whose hidden thinking spends from the same output budget.

Constraints the schema silently ignores. Providers enforce types, enums, and required keys — but commonly not string lengths or numeric ranges. Claude's structured outputs, for instance, don't support maxLength or minimum; the SDKs strip those keywords and leave validation to the client. So a rule like "an App Store subtitle is at most 30 characters" cannot live in the schema. It has to live in your code, after the parse.

Values that are wrong. The schema guarantees a string where a string belongs. It has no opinion on whether that string is keyword spam, whether the bounding box sits outside the image, or whether the "confidence" is a hallucinated 0.99. Shape is not correctness — semantic validation stays your job, and the good news is that a guaranteed shape makes it easy to write.

Field notes from production

ShotCanvas leans on all of this daily. Our focal-point detector sends a screenshot to Gemini with a five-line responseSchema — a boolean and an array of integers — and it has been boringly reliable: small closed schema, low temperature, nothing to strip. That's the happy path, and most extraction tasks can live on it.

Our vision benchmarking path taught us the caveat. On requests that mix text and images, we found the JSON-output flag was frequently ignored — the model returned prose around the object anyway — so that code path deliberately skips JSON mode and keeps the fence-stripping parser, with a retry that falls through to the next model on our list. There's a comment in the source warning future-us not to "fix" it. The lesson generalizes: structured output is a strong guarantee, not a universal one. Keep the cheap fallback parser, keep a retry path, and log the raw response on failure so you're debugging evidence instead of vibes.

And the metadata generator — the thing writing store descriptions and subtitles — clamps every field in code after parsing, because Apple's 30-character subtitle limit and 170-character promo limit are exactly the kind of rule schemas won't enforce. The model is asked politely, the schema guarantees the keys, and the last line of defense is a substring.

Where we landed

Every AI feature in ShotCanvas — the headline writer, the auto-designer, the metadata generator — sits on this stack: schemas where they hold, fallback parsing where they don't, and hard clamps where the app stores get the final vote. If you'd rather see the output than the plumbing, the generator is a click away.