Why json_object is the Beginning, Not the End

In the first week of deployment, the function that parses settlement records died. The logs showed a JSONDecodeError, and right next to it, finish_reason was length. This was despite having json_object set in response_format. The JSON came out cut off mid-way, and a truncated JSON is nothing more than a string, no matter how intact it looks. The caller was promised 'valid JSON', not 'complete JSON'.

The only promise json_object keeps is syntactic

response_format: { "type": "json_object" } guarantees exactly one thing: that the returned string parses under JSON grammar. It doesn't look at schemas. A missing key, a null in place of an array, a date arriving as a number instead of a string—all of that is still valid JSON. The structure is decided probabilistically each time. It might rename a field, or append a key that was never requested. Conversely, if you skip this mode entirely, the model is prone to wrapping JSON in a code fence or appending explanation text after the JSON.

There are also constraints thrown in for free. Using this mode means the word "json" must appear somewhere in the conversation. If it doesn't, the call itself is rejected with a 400. More precisely, you get an error like messages with role 'system' must contain the word 'json'. It's common for an existing prompt to hit this wall when you only meant to add formatting. Model support also differs. json_object covers even relatively older models, but the json_schema mode, which enforces a schema directly, has its own support list and can't be used with certain fine-tuned models. From the start, you need to branch on 'what does this model support?'.

There are three places where it breaks

Runtime failures divide roughly into three layers.

Parsing failure is the first. This is when the syntax itself is broken. If the output is truncated by max_tokens, it dies at this point almost without exception. For a service that receives responses where a single object spans thousands of characters, truncation is an accident that can happen at any time.

Schema failure is the second. The JSON is fine, but a required key is missing or a type doesn't match. Whether a KeyError fires because a field is entirely absent, or an AttributeError fires because null came through, depends on the moment.

The third is semantic failure. It passes the schema completely, but the values don't make sense. A price comes in negative, or a product code arrives as a value not in the allowed list. This is not caught by JSON Schema. A schema only reviews shape; it doesn't check whether a value falls within the range allowed by the domain. In the end, validation has to pass through the schema layer and the semantic layer separately, and the last layer stays in the domain logic. If you handle both layers with a single validation layer, semantic validation tends to get trapped inside the expressive range of the schema library.

A loop that turns failure into feedback

If you end a validation failure as an error, that's where it stops. The core of this safety net is feeding the failure message into the next call and asking again. If validation is the first line of defense, retry is the second. Responses that get caught at the parsing or schema stage have a fairly good chance of being rescued by a retry. With truncation in particular, asking again with a higher max_tokens often completes at the same spot.

def request_json(messages, schema_validator, max_retries=2, **kwargs):
    for attempt in range(max_retries + 1):
        resp = client.chat.completions.create(
            model=MODEL,
            messages=messages,
            response_format={"type": "json_object"},
            **kwargs,
        )
        finish = resp.choices[0].finish_reason
        try:
            data = json.loads(resp.choices[0].message.content)
            schema_validator(data)
            return data
        except (json.JSONDecodeError, ValidationError) as e:
            messages = [
                *messages,
                {"role": "assistant", "content": resp.choices[0].message.content},
                {"role": "user",
                 "content": f"The previous response was not valid JSON or violated the schema: {e}. Follow the schema and output only JSON again."},
            ]
    raise MaxRetryExceeded(finish_reason=finish)

Retry feedback needs to be specific for the loop to close quickly. Guidance like "the JSON is wrong" is useless. You have to hand over which key came in as what—"the price field came as a string; it should be a number"—for the next attempt to differ. Conversely, if you paste the whole error message and just say "try again," the same mistake repeats.

There is one exception. If it died with finish_reason set to length, you should check max_tokens before adding feedback. Retrying without fixing the cause of truncation only burns the budget.

If it keeps getting caught at the semantic stage, you shouldn't try to solve it with retries. That failure doesn't come from the call loop; it comes from ambiguity in the prompt or the schema definition. Asking for the same content two or three times won't change the values. At that point, you fix it by adding an enum to the schema or putting examples in the prompt.

When you move up to json_schema

If the model supports it, switching the type in response_format from json_object to json_schema with strict: true gives more solid results. Because the sampling itself is forced to stay tied to the schema, the rate of schema violations drops significantly. But there's no exemption. The official docs state that you should still validate the output even in this mode. Writing schemas for strict mode comes with constraints too. It requires additionalProperties: false, every property has to be in the required list, and only a subset of JSON Schema keywords is allowed. Shoving in an existing schema as-is just produces errors. A strict schema is work you rewrite to fit the rules, not a copy-paste.

If the model doesn't support it, you drop back down to json_object. In other words, this safety net always has to be attached to either one. The stronger the schema enforcement, the lower the failure probability, but there is no moment where it reaches zero. Removing the parse-validate-retry loop because you're using json_schema doesn't hold up.

The retry budget and the last line of defense

Retries aren't free. Each call adds a few seconds of latency and token cost. Without a cap, a single rare failure can swallow the entire latency budget. Two retries is the practical number. Allowing three or more pushes the tail of the response-time distribution out of control. The budget criteria differ by situation. For a real-time response path, cap the total elapsed time in seconds; for a batch pipeline, a retry count cap is enough. The latter prioritizes throughput over latency.

If it fails after exhausting the budget, return a default object or hand it off to a manual review queue. Either way, don't write code that looks like it's swallowing the failure. If you return an empty object and move on quietly, truncated JSON piles up in the background. The logs should record finish_reason, the failure point (parse/schema/semantic), and the attempt count, so you can find the cause later.

Look at the change in numbers, not pass/fail

There's exactly one way to know before deploy whether this safety net actually works: pick a few dozen representative prompts and measure the failure rate. The values to watch are three: first-attempt failure rate, retry recovery rate, and budget-exhaustion ratio. Record these three numbers and compare them every time you change the model or modify a prompt. The validation set must include truncation-inducing cases, like long product descriptions or responses that send down multiple items. You can't reproduce truncation with unit tests. You need stress runs that fire the same prompt multiple times. For example, if the budget-exhaustion ratio rises even one percentage point from before, it's right to inspect the prompt and schema before adjusting the retry settings.

json_object is a starting point, not a destination. Passing JSON through without validation is like scheduling your own night where parsing errors blow up even though the mode is on.

Comments

Loading comments.