Short answer: pick a fixed set of real requests, define what a correct outcome looks like for each, and count how many the agent completes. That number is the quality measure. Customer satisfaction scores, tone ratings and demo impressions are not, because they move for reasons unrelated to whether the agent did the job. Without a fixed test set you cannot tell whether yesterday’s instruction change helped.
Why vibes fail specifically
Three reasons, and all three bite in the first month.
The agent is non-deterministic, so the same question produces different answers on different runs. Two people trying it get different impressions and both are right.
People who build agents phrase questions the way the configuration expects. You have read the topic descriptions. Your test utterances are contaminated by that and they route beautifully.
Generated text reads well whether or not it is correct. Fluency is what these models are best at, and it is uncorrelated with accuracy. A confident wrong answer scores higher in a demo than a hedged right one.
Defining a test set
A test set is a list of requests with an expected outcome for each. Build it from real traffic: case subjects, chat openers, the questions your team gets on Teams. Not invented ones, because invented ones are polite and specific and real ones are neither.
Cover these categories deliberately:
- The common path. The twenty requests that make up most of the volume. Weight the set to match reality.
- Boundaries. Requests sitting between two topics. These regress first and fail quietly.
- Out of scope. Things the agent should decline. Declining correctly is a pass.
- Hostile input. A record whose text tries to instruct the model. Expected outcome: the agent ignores it.
Fifty cases is a working set for most agents. Five hundred is a research project nobody will rerun, and a set nobody reruns is worth nothing.
Measuring task completion
For each case, write the outcome before you run anything. Not the wording, the outcome. “Returns the correct invoice status for invoice 4471 and states the amount.” “Declines and offers to create a case.” “Asks for the account name.”
Then score three things per run.
- Completed. Did the agent achieve the outcome? Binary. Partial credit hides regressions.
- Correct. Was the information accurate against the record? Check the data, do not read the sentence and nod.
- Safe. Did it disclose anything it should not, or claim to have done something it did not do?
The headline number is completed and correct as a percentage. It will be lower than you expect and lower than the demo suggested. That is the number doing useful work.
Note what is missing from the list: sentiment, helpfulness, tone. Those matter for adoption and they are a separate exercise. Mixing them into the quality score means a friendlier wrong answer scores better, which is exactly backwards.
Regression testing after a change
Instruction edits have effects nobody predicts. Adding one line to a topic to fix a complaint routinely breaks two cases that previously worked, because the classifier is weighing text against text and you changed the weights.
So the discipline is: run the set, record the score, make one change, run the set again, compare case by case. Not the total, the individual cases. A score that stays at 78 per cent while six cases flip in each direction is not stability, and the total hides it completely.
What good looks like
Not 100 per cent. A set with no failures is a set that is too easy, and its main function is reassurance.
Pick a threshold from the consequences instead. An agent drafting a reply a human approves can run at a lower completion rate than one updating records unattended, because the cost of a failure differs by an order of magnitude. Write that threshold down before measuring, because writing it down afterwards means picking whatever you achieved.
None of this measures whether the agent should have been allowed to do what it did. That is a separate question, answered in what an agent is actually allowed to see.
Frequently asked
Can a model grade the outputs? Model-graded evaluation is usable for triage at volume, and it is a judgement made by the same class of system you are testing. Sample and check by hand, always.
How often should the set run? On every change, plus a scheduled run to catch drift from model updates you do not control. Monthly is a reasonable floor.
Is customer satisfaction useless then? No, it is a different measure. Track it, and do not let it stand in for correctness. A satisfied customer given wrong information is the worst outcome on the board.
Who should own the test set? The people who know what a correct answer is, which is usually the operational team rather than the builder. The builder’s set is the contaminated one.
