Reliable Structured Data from an LLM: Schemas and Retries — AI Automation article by Rui Codex

Reliable structured data from an LLM comes from combining a strict JSON schema, native structured-output support where your provider offers it, validation of every response in code, and targeted retries that send the specific validation error back to the model. Retries alone rarely fix a poorly defined schema.

Key takeaways

  • A schema is the contract between your code and the model, so define it before you write the prompt.
  • Validate every response in code, because a model can return well-formed JSON that still breaks your business rules.
  • Retry only on failures a retry can plausibly fix, and include the validation error in the retry request.
  • Cap retries and log every final failure, because an unbounded retry loop hides faults and wastes capacity.

What does a JSON schema actually guarantee?

A JSON schema describes the keys, types, required fields and allowed values a response must have. It guarantees shape, not truth. A value can pass every schema rule and still be wrong.

JSON itself is defined in RFC 8259, the IETF specification for the data interchange format. The vocabulary used to describe structure is documented by the JSON Schema project, which is the usual reference when you write schemas by hand.

The trade-off is strictness. A schema that is too tight, with many required fields and narrow patterns, causes more failed responses. A schema that is too loose lets incomplete or malformed values through to your database. Start with the fields your downstream process truly needs, then tighten the schema only where a bad value would cause a real error.

Should you use native structured output or prompting alone?

Use native structured output when your provider supports the schema features you need, because the response is constrained during generation. Fall back to prompt-only JSON with strict validation when the feature is unavailable or the schema uses constructs the provider does not support. The OpenAI Structured Outputs documentation describes how constrained decoding works and which schema features are supported, so check that page against your schema before you commit to it.

ApproachStrengthTypical failure mode
Native structured outputOutput conforms to the schema by constructionUnsupported schema features, and refusals that return no usable object
Prompt-only JSONWorks with any model and needs no special setupMarkdown fences, trailing commentary, unescaped quotes, truncated output
Tool or function callingArguments are described by a schema the model is trained to followArguments that are valid JSON but semantically wrong for the task

In all three cases the output still needs validation in your own code. Native constraints reduce parsing errors, but they do not check your business rules.

How should you validate model output?

Validate in three stages, in order: parse, check the schema, then check business rules. Each stage catches a different class of error, and each needs a different response.

  1. Parse the raw text as JSON. If parsing fails, strip code fences and surrounding prose, then try again before counting it as an error.
  2. Validate the parsed object against the JSON schema. Record the path of every failing field, because that message is what you will send back on a retry.
  3. Run business rules that a schema cannot express, such as a total that must equal the sum of line items, or an end date that must follow a start date.
Tip: Log the raw response next to the validation error. Most schema failures repeat in patterns, and the raw text shows you whether the fault is in the prompt, the schema or the input document.

Checklist before going live

  • Every response path reaches a parse, a schema check and a rule check, with no silent pass-through.
  • Truncated output is detected, for example by a stop reason that is not a normal completion, and treated as a failure.
  • Failed records go to a review queue rather than being written to the system of record.
  • Each validation rule has a test case with a known bad input.

Teams that build this layer as part of a wider data flow often connect it to their existing systems. Our integration services page describes how validated records move between systems in practice.

When do retries help and when do they make things worse?

Retries help when the failure is transient, or when the validation error is specific enough for the model to correct, such as a missing field or a date in the wrong format. They make things worse when the schema or the input is the real problem, because the same failure repeats on every attempt.

Separate two kinds of retry. Transport retries, such as timeouts or rate-limit responses, should use exponential backoff and a short cap. Content retries should be repair attempts: send the original input, the invalid output and the exact validation message, then ask for a corrected object.

  • Keep the retry count fixed and small, and include it in your logs.
  • Lower the sampling randomness for repair attempts, so the model is less likely to drift.
  • Do not retry on refusals or empty responses without changing something; log them and route them for review.
  • Stop after the cap and record the final error with the input identifier, so a person can decide what happens next.

How do you design a schema a model can follow?

Keep the schema flat, name each field precisely and make absence explicit. A model finds a clear, small object easier to fill correctly than a deep tree with overlapping fields.

  • Use enumerations for categories instead of free text, and include an explicit other value with a required note.
  • Describe each field in plain language, because the description is part of the instruction the model reads.
  • Decide whether a missing value should be null or omitted, and state that rule once for every field.
  • Split very large extractions into smaller schemas, so one failure does not discard the whole document.

If you are unsure how to scope a schema for a particular process, our AI automation service page outlines how automated extraction and decision steps are usually structured.

Hypothetical examples

Here are two hypothetical scenarios:

Invoice extraction. A schema requires an ISO date for the invoice date. The model sometimes returns a day-first format. Schema validation catches it, the repair attempt includes the message naming the field, and the corrected date passes. The total-versus-line-items mismatch is different: it is a business rule, so it goes to review instead of being retried.

Ticket classification. A schema restricts the category to a fixed list. The model keeps returning a category name that is not on the list. Retrying does not help here, because the categories do not match how the team actually works. The fix is to revise the enumeration and its descriptions, add an explicit other value, and route those tickets to a person.

Frequently Asked Questions

Does a JSON schema make an LLM's answers correct?

No. A JSON schema checks shape: which keys exist, their types and their allowed values. It cannot confirm that a value is true, so business rules and cross-field checks still need code after the schema check passes. Treat schema validation as the first gate, not the final one.

Should I use native structured output or ask for JSON in the prompt?

Use native structured output when your provider supports the schema features you need, because the response is constrained during generation. Prompt-only JSON is acceptable for prototypes, but you must still validate every response, since code fences, trailing commentary and truncation are common failure modes.

How many retries should a pipeline allow?

Keep the number small and fixed, usually one or two repair attempts for content errors, with a cap on total attempts per request. Include the validation error in each retry, and stop once the cap is reached so the failure is logged and routed to a person or a queue.

Can a retry fix a schema that is wrong?

Usually not. If the schema demands information the input does not contain, or uses a type the process never needs, the model fails in the same way on every attempt. Review the failure log, adjust the schema or the input, and only then retry.

When should I bring in outside help for this kind of pipeline?

Build the extraction layer yourself when the output feeds one well-defined process. Outside help is worth considering when the schema must match several systems, such as an ERP and an e-commerce platform, and the validation rules need to reflect how your data is actually stored. You can request a project consultation at the Rui Codex contact page.

Conclusion

Reliable structured data from an LLM is an engineering problem with a clear shape. Define a schema that describes only what your process needs, validate every response in stages, and use retries only where a corrected prompt can plausibly fix the fault. Log everything that still fails, and review those failures to improve the schema and the input rather than adding more attempts. Request a project consultation at https://ruicodex.com/Contact, or email info@ruicodex.com to discuss your project.

Sources

Tags: JSON Schema LLM output validation structured output AI automation API integration retry logic prompt engineering data quality

Need Help Implementing This?

Our team can help you put these insights into practice. From AI automation to custom software development, we build solutions that deliver real results.

Send a project request