Pick one use case, one audience of about twenty people, and run it for four weeks with someone reading transcripts every day. That is the whole plan. Big-bang launches fail not because the agent is bad but because nobody discovers how it is bad until several thousand customers have already met it, and by then the organisational verdict is fixed.
Before the pilot
Three things have to exist first, and skipping any of them makes the pilot unmeasurable.
A test set. Thirty to fifty real requests, taken from actual transcripts or cases, each with a written statement of what a correct outcome looks like. Not invented by the build team, because the build team phrases things the way the configuration expects.
A baseline. How long does this work take today, how often is it resolved first time, what does it cost. Without it every later claim of improvement is unfalsifiable, and someone will eventually ask.
A named owner. One person accountable for the agent’s behaviour, with the authority to deactivate it. Not a committee.
The pilot group
Twenty to fifty people, chosen deliberately. Internal agents first if you have a choice, because internal users give you blunt feedback and tolerate rough edges. For a customer-facing agent, a controlled slice: one region, one product line, one entry point, ideally outside your highest-value segment.
Who to include: a couple of sceptics, a couple of people who will use it constantly, and someone experienced enough to notice a subtly wrong answer. Enthusiasts-only pilots tell you nothing, because enthusiasts work around problems and do not report them.
Tell them it is a pilot, that it will get things wrong, and exactly how to report a bad answer. Make reporting take under thirty seconds. A form nobody fills in produces a pilot with no findings and a false sense of success.
What to measure during it
Weekly, and keep it short:
- Test set pass rate. Rerun the whole set after every change. This is the only number that tells you whether a fix helped or just moved the problem.
- Transcripts read. Yes, as a metric. If the count is zero for a week, the pilot has stopped being a pilot.
- Escalation rate, split by reason. Repeated failure is the one to drive down. User requested may be perfectly healthy.
- Repeat contact within seven days. The best available proxy for whether anything was actually resolved.
- Action failure rate per action. Flows throw errors and agents apologise vaguely. Monitor the actions directly.
Run the test set repeatedly rather than once, because behaviour is non-deterministic. A single clean pass proves nothing about a distribution.
Why big-bang launches fail
The failure modes are consistent enough to list.
The volume of edge cases. Real users produce phrasings nobody anticipated, and at scale they arrive faster than you can respond. A pilot surfaces the same cases at a rate a team can work through.
No baseline to argue from. Six weeks in, someone asks whether it is working. Without a before-and-after you get an argument about anecdotes, and the loudest anecdote is always a bad one.
Reputation sets early. If the first thing a thousand agents-in-the-field experience is a bad answer, you will not recover that audience with a later fix. Internal opinion about AI tools hardens fast.
Nobody reads anything. At pilot scale a person reads every transcript. At full scale nobody does, and problems become invisible precisely when they matter.
Expanding
Widen on evidence, not on the calendar. Reasonable gates: test set pass rate stable across two weeks, repeated-failure escalations trending down, no unresolved severity-one findings, and the owner willing to say it is ready.
Expand one dimension at a time. More users, or more topics, or a new channel. Changing two at once means a regression tells you nothing about its cause.
After launch
The agent is not finished. The model underneath is updated on a schedule you do not control, your knowledge content changes, and your customers’ phrasing shifts. Rerun the test set monthly regardless of whether anything on your side changed. That single habit catches the class of regression nobody else will notice.
Frequently asked
How long should a pilot run? Four to six weeks. Shorter misses the second-order problems. Longer loses momentum and the team stops paying attention.
What if the pilot goes badly? That is a successful pilot. Decide whether the use case was wrong or the build was, and be honest about the first, because a bad use case cannot be fixed by better instructions.
Should we announce it internally? Quietly during the pilot. A launch announcement creates expectations the pilot version cannot meet.
Can we skip the pilot for an internal tool? You can. You will discover the same problems with a larger and less patient audience.
