Testing Center runs a batch of utterances through your agent and checks what it did, not how nicely it phrased the reply. The assertions that matter are which topic was selected and which action was called. That catches routing regressions, which are the most common way an agent breaks. It tells you very little about whether the conversation was any good, so manual testing does not go away.
What a test case is
A test case is an utterance plus the expected outcome: the topic that should be chosen, and usually the action that should fire. You supply a set of these, run them as a batch, and get a pass or fail per row with the actual topic and action shown next to the expected one.
That is a narrower thing than it first sounds, and the narrowness is the point. It is a routing test. The agent’s free text response is generated fresh each run and is not a stable thing to assert on, so the framework mostly asserts on the structured decisions underneath it.
Generating test cases
Salesforce can generate candidate utterances from your topics and instructions. Use it, then edit heavily.
Generated cases inherit the vocabulary of your own configuration. They ask the question the way the topic description phrases it, which means the agent routes them correctly for the wrong reason. A test set that only contains well-formed questions in your own words will pass forever and warn you about nothing.
The cases worth adding by hand:
- Real utterances. Pull them from chat transcripts, case subjects, or whatever your support queue already contains. Misspelt, truncated, angry.
- Boundary cases. Questions that sit between two topics. These are the ones that regress.
- Out of scope questions. Assert that the agent declines, or routes to a fallback, rather than improvising.
- Follow-ups. “And the other one?” makes no sense without history, and that is exactly the shape of real second turns.
What an agent test actually asserts
Be honest about the claim a green run supports. It says: for these inputs, the classifier picked the topic I expected and the planner called the action I expected. It does not say the action returned correct data, that the wording was appropriate, that nothing sensitive was disclosed, or that a customer would have been satisfied.
Some releases add checks over the response text as well. Where those exist they are still a judgement made by a model on generated text, which is useful as a signal and weak as a gate. Read the current documentation for what your org supports before designing a pipeline around it.
Why manual conversation testing is still needed
Batch tests are single turns. Almost every real problem in a deployed agent is multi-turn: the agent loses the record it was working with, re-asks a question it already had the answer to, confirms an update it never performed, or drifts out of the topic it started in and never comes back.
None of that is visible in a one-utterance test. Sit down and have ten complete conversations before any pilot, and have someone who did not build the agent do the same, because you unconsciously phrase everything the way the configuration expects.
Using it as a regression gate
The real value arrives on the second run. Change one instruction, rerun the set, and see which previously passing cases now fail. Instruction edits have effects nobody predicts, and this is the only cheap way to see them.
Practical habits:
- Run the full set before and after every instruction or topic change, not weekly.
- Keep the set in version control alongside the agent metadata so a change and its test results are reviewable together.
- Expect some flakiness. Generated systems are not deterministic, and a case that fails one run in ten is information, not noise.
Frequently asked
How many test cases are enough? Enough to cover every topic and every boundary between adjacent topics. For a ten-topic agent that is typically a few dozen, weighted towards the pairs that get confused.
Can this run in CI? Tooling for scripted runs has been appearing through the CLI, and what is available changes by release. Check the current CLI documentation rather than assuming.
Does a passing set mean the agent is ready? It means routing is behaving. Readiness also needs permission review, action-level testing, and real conversations with real people.
Should tests run against production? Test where the configuration is, which is normally a sandbox. Running a batch against a production agent consumes the same capacity real conversations do, and that shows up on the bill.
