Skip to main content
A run gives each call one pass or fail. Give each step of the call its own custom metric, so every run also shows which step failed and on which calls.

​
Before you start

  • Create a simulation that covers the flow you want to protect. See Create a Simulation.
  • Save the current agent prompt as an agent version, so that you have a baseline to compare against.

​
Steps

1

Write one metric for each check

Create a custom metric for each step you always care about, for example that the agent quoted the right price, booked the appointment, and captured a callback number. Give each metric a single question and the Pass/Fail response type, so a failure points to one step. Write the prompt so it names what counts as a pass and what counts as a fail.The Create Metric dialog with a Pass/Fail metric that checks whether the agent books the appointmentThe Create Metric dialog with a Pass/Fail metric that checks whether the agent books the appointmentOpen the simulation and add the metrics in its Custom Metrics tab.The Custom Metrics tab of a simulation with three Pass/Fail metrics enabled: Stays polite, Verifies identity first, and Greets the callerThe Custom Metrics tab of a simulation with three Pass/Fail metrics enabled: Stays polite, Verifies identity first, and Greets the caller
2

Check the important metrics in Metrics Lab

A metric that judges wrongly makes a regression look real or hides one. Run your key metrics in Metrics Lab against calls you have already judged yourself, and fix any that disagree. See Improve the Accuracy of a Metric.The Lab tab on the Metrics page, listing labs with their conversation and metric countsThe Lab tab on the Metrics page, listing labs with their conversation and metric counts
3

Run the simulation after each prompt change

Save each prompt change as a new agent version. On the simulation, click Create New Run and choose that version in Agent version, so each run belongs to one known version of the agent. Change one thing per version, so a regression points to one change.The Create New Run dialog on the Version Settings tab with the Agent version list open, showing Version 7 as Production and earlier versionsThe Create New Run dialog on the Version Settings tab with the Agent version list open, showing Version 7 as Production and earlier versions
4

Compare the metrics across runs

In Run History, select the run before the change and the run after it. Click Compare Runs, then Compare.Run History with two runs checked and the Compare Runs button enabled, one run on Version 3 and one on Version 5Run History with two runs checked and the Compare Runs button enabled, one run on Version 3 and one on Version 5In the dialog, Compare by starts on Digital human outcomes. The header counts the regressed Digital Humans, and the matrix marks each one that went from Pass to Fail.The Compare Runs dialog comparing two runs by Digital human outcomes, with 3 regressed and the Digital Humans that went from Pass to Fail marked with a down arrowThe Compare Runs dialog comparing two runs by Digital human outcomes, with 3 regressed and the Digital Humans that went from Pass to Fail marked with a down arrowThen set Compare by to Custom metrics passed (summary) to see which runs dropped, and to each metric in turn. The metric whose calls went from Pass to Fail is the step that broke.
5

Open a failing call

Open a Digital Human that regressed and read the failure reason, then the transcript, to see what the agent said at that step. Use that to decide the prompt fix, save it as a new agent version, and repeat from the run step.A simulation run with the Conversation Details panel open on a failed call, showing the success criteria and the failure reasonA simulation run with the Conversation Details panel open on a failed call, showing the success criteria and the failure reason

​
Check the result

When the setup works, a drop in the pass rate traces to one or two metrics that regressed, and you can open the failing calls to see what the agent said at that step. If every metric still passes while the pass rate drops, the failure is in a step that no metric checks yet, so add a metric for it. Some checks apply only to some calls, for example a callback number only matters when the agent offered a callback. Make that metric conditional, so it runs only when the agent called a given tool or the call reached a given workflow node. On other calls it does not run and has no result, so it does not drag down the comparison. See Conditional metrics.
Last modified on October 22, 2026