How to Test an LLM Automation Before Go-Live with Test Sets — AI Automation article by Rui Codex

To test an LLM automation before go-live, build a fixed test set of realistic inputs with known expected outcomes, define pass criteria for each task before you run anything, score the outputs against those criteria, and release only when the failures stay within limits agreed in advance.

Key takeaways

  • A test set is a fixed, versioned collection of realistic inputs paired with expected outcomes, and it is reused for every change.
  • Pass criteria should be written for each task before the first run, so the results cannot quietly reset the standard.
  • Critical failures, such as wrong amounts or unauthorised promises, should block release regardless of the overall score.
  • Personal data in test environments must be justified and minimised under the GDPR.

Why does an LLM automation need a fixed test set?

A fixed test set lets you measure the same workflow the same way before and after every change. Without one, teams judge outputs from a few memorable examples, and those examples usually look good because someone chose them.

LLM behaviour shifts with the prompt, the model version, the retrieved documents and the sampling settings. A wording change that fixes one case can quietly break three others, and you only see that when the same cases run again. For background on AI workflow design, see the AI and workflow automation guides on the Rui Codex blog.

Test the whole pipeline, not just the model reply. Most automations are a chain: the prompt is built, the model is called, the output is parsed and validated, and the result is written to a business system through system integrations that connect your ERP and other tools. Each step can fail, and a test that only reads the model text will miss a broken field mapping.

Tip: Version the test set together with the prompt and workflow configuration, so every result can be traced to the exact combination that produced it.

How do you build a useful test set?

Build the set from realistic inputs taken from the process you are automating, and pair each input with an expected result written by someone who knows that process. Start with the cases your operators handle every week, then add the unusual ones.

Where the cases come from

Historical tickets, emails, documents and records are the most representative source. Synthetic inputs are easier to share and avoid personal data, but they tend to be cleaner than real inputs, so they miss the typos, missing fields and forwarded threads that break automations.

How to write expected outcomes

An expected outcome is the result a competent operator would accept, not a perfect transcript. For extraction tasks, write the exact values. For drafting tasks, write the facts that must appear and the claims that must never appear.

Coverage checklist

  • Typical cases for each task type
  • Incomplete or contradictory input
  • Every language and document format the business actually receives
  • Edge cases that previously needed a manual correction
  • Adversarial inputs, such as instructions hidden inside a customer message

The trade-off is size against maintenance. A large set takes longer to label and to rerun, while a small set can hide weaknesses. Grow the set from production incidents so that every real failure becomes a permanent test.

What should pass criteria measure?

Pass criteria should measure whether each output is safe to use for the next action, and they should be checkable without debate. Write them for each task before the first run, so the results cannot shape the standard.

Use three kinds of check. Deterministic checks are rules, such as valid JSON, a required field present, or a value that exists in an allowed list. Rubric checks are scored by a human reviewer against written guidance. Model-judge checks use a second model to score a rubric, which scales well but is trustworthy only after its scores have been compared with human labels on the same cases.

Check typeWorks well forWhere it fails
Deterministic rulesFormat, required fields, allowed valuesCannot tell whether a correct-looking answer is actually right
Human rubricTone, completeness, nuanced judgementSlow, and reviewers drift apart unless they are calibrated
Model judgeHigh-volume scoring of subjective rubricsShares the blind spots of the model it judges and must be calibrated

Classify failures by severity. A wrong bank account or a promised refund is a critical failure that blocks release, whereas an awkward sentence is a minor failure that can be logged and fixed later. Set the release threshold in advance, and for critical failures it should be zero.

How do you test an LLM automation before go-live?

Run the complete pipeline against the frozen test set, score every output with the agreed checks, and group the failures by cause before you change anything. Changing the prompt right after a failure is the most common way teams lose track of what actually improved.

  1. Freeze the test set, the prompt and the configuration under one version label.
  2. Run the full set, and repeat runs where outputs vary, to see how stable each result is.
  3. Score each output against the criteria, recording the reason for every failure.
  4. Group failures by cause: retrieval, prompt, parsing, model behaviour or source data.
  5. Fix one cause, then rerun the entire set rather than only the failing cases.

Repeated runs cost more time and model usage, and that is a trade-off worth making wherever a single bad output causes harm. The main failure mode of this process is overfitting: if the same cases guide every tweak, the set stops measuring new behaviour. Keep a held-out subset that is opened only for the final decision. Testing the live connections is a separate step, and it is covered by Rui Codex AI automation services.

Can you use real customer data in a test set?

Yes, but only when there is a clear purpose, a lawful basis and a reason to keep personal data outside production. Remove or pseudonymise identifiers wherever the test still works without them.

The General Data Protection Regulation sets out principles for processing personal data, including lawfulness, purpose limitation and data minimisation, as stated in Article 5 of Regulation (EU) 2016/679. This article is general information and not legal or tax advice.

Practical controls include restricting access to the test environment, deleting copies once testing ends, and documenting why each dataset is there. Rui Codex offers technical implementation support for these workflows, while questions about legal obligations belong with your counsel.

Hypothetical examples

Here are two hypothetical scenarios:

In the first, a workflow extracts supplier invoice fields from incoming emails and PDF attachments. The critical checks require exact amounts and bank details that match the source document. Failures cluster in low-contrast scanned PDFs, so the fix is an input validation step with a rejection path rather than a prompt rewrite.

In the second, a drafting assistant writes first replies to customer support messages. Its criteria forbid any promise about refunds or delivery dates, and the test set includes messages that pressure the assistant into making one. A draft that promises a refund blocks release, and the remedy is a routing rule that sends refund requests to a person.

When is an LLM automation ready to go live?

It is ready when every critical check passes on the frozen test set, the remaining failures are understood and accepted in writing, and a rollback path has been tested. If any one of these is missing, the release rests on hope rather than evidence.

  • Critical checks pass on the frozen set with no exceptions
  • Held-out cases were run once, after the final change
  • Known minor failures are logged with an owner and a fix date
  • A human review step exists for actions that cannot be undone
  • Rollback to the previous process has been rehearsed

A staged rollout reduces risk further. Run the automation in shadow mode first, where outputs are compared with human decisions but not acted on, and only then let it write to live systems.

Frequently Asked Questions

How many test cases does an LLM automation need before go-live?

There is no universal minimum, because the right size depends on the number of task types, input sources and known failure modes. The set must cover each of these at least once, and it should keep growing from production incidents. A smaller set of well-chosen cases is more useful than a large set of near-duplicates.

Can an LLM judge replace human reviewers when scoring outputs?

An LLM judge can score subjective rubrics at volume, but it should not replace human review until its scores agree with human labels on the same cases. Use deterministic rules for anything a rule can check, and review every case where the judge and a human disagree. Remember that the judge can share the weaknesses of the model it is scoring.

What is the difference between a test set and a development set?

A development set is used while you tune prompts, rules and parsing, whereas a test set is frozen and used only to decide whether the automation can go live. Keep the two separate so the final result still measures behaviour the team has not optimised for. Cases used during tuning should not count toward the release decision.

Is one successful test run enough before deploying?

No, one successful run only shows that the system behaved well once, for one version of the inputs. Models, retrieved content and upstream data change over time, so rerun the frozen set after every change and monitor production outputs against the same criteria.

Conclusion

A test set turns go-live from a judgement call into a documented decision. Build it from realistic inputs, write pass criteria before the first run, separate critical failures from minor ones, and keep a held-out subset for the final check. Revisit the set after every production incident, because the most useful test is usually the failure you did not expect. If you want an engineering review of your test design before release, you can request a project consultation.

Sources

Tags: LLM testing test sets pass criteria AI automation workflow automation go-live checklist GDPR regression testing

Need Help Implementing This?

Our team can help you put these insights into practice. From AI automation to custom software development, we build solutions that deliver real results.

Send a project request