> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Find Which Check a Regression Broke

> Find the step that broke after an agent change.

A run gives each call one pass or fail. Give each step of the call its own [custom metric](/test/metrics.md), so every run also shows which step failed and on which calls.

## Before you start

* Create a simulation that covers the flow you want to protect. See [Create a Simulation](/test/simulations/create.md).
* Save the current agent prompt as an [agent version](/test/simulations/reference.md#agent-versions), so that you have a baseline to compare against.

## Steps

**1. Write one metric for each check**

Create a [custom metric](/test/metrics.md) for each step you always care about, for example that the agent quoted the right price, booked the appointment, and captured a callback number. Give each metric a single question and the **Pass/Fail** response type, so a failure points to one step. Write the prompt so it names what counts as a pass and what counts as a fail.

![The Create Metric dialog with a Pass/Fail metric that checks whether the agent books the appointment](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metric-create-dialog.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=859613c65313758fa8501964443f47e5)

Open the simulation and add the metrics in its **Custom Metrics** tab.

![The Custom Metrics tab of a simulation with three Pass/Fail metrics enabled: Stays polite, Verifies identity first, and Greets the caller](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-find-regression-sim-custom-metrics.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=3d4b8eeae23d99002e1c29b723cba217)

**2. Check the important metrics in Metrics Lab**

A metric that judges wrongly makes a regression look real or hides one. Run your key metrics in Metrics Lab against calls you have already judged yourself, and fix any that disagree. See [Improve the Accuracy of a Metric](/cookbook/improve-metric-accuracy.md).

![The Lab tab on the Metrics page, listing labs with their conversation and metric counts](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-landing.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=13d692dc70c449b65fedc5dc9f4e6c21)

**3. Run the simulation after each prompt change**

Save each prompt change as a new [agent version](/test/simulations/reference.md#agent-versions). On the simulation, click **Create New Run** and choose that version in **Agent version**, so each run belongs to one known version of the agent. Change one thing per version, so a regression points to one change.

![The Create New Run dialog on the Version Settings tab with the Agent version list open, showing Version 7 as Production and earlier versions](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-find-regression-create-run.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=d4642493483cd0c3c952aed4e07d72dc)

**4. Compare the metrics across runs**

In **Run History**, select the run before the change and the run after it. Click **Compare Runs**, then **Compare**.

![Run History with two runs checked and the Compare Runs button enabled, one run on Version 3 and one on Version 5](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-find-regression-run-history.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=5a207bb09c0e39271bc32d010a7d1716)

In the dialog, **Compare by** starts on **Digital human outcomes**. The header counts the regressed Digital Humans, and the matrix marks each one that went from Pass to Fail.

![The Compare Runs dialog comparing two runs by Digital human outcomes, with 3 regressed and the Digital Humans that went from Pass to Fail marked with a down arrow](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-find-regression-compare-dialog.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=bedfd903061668c58ec6fb8e56fe0b1b)

Then set **Compare by** to **Custom metrics passed (summary)** to see which runs dropped, and to each metric in turn. The metric whose calls went from Pass to Fail is the step that broke.

**5. Open a failing call**

Open a Digital Human that regressed and read the failure reason, then the transcript, to see what the agent said at that step. Use that to decide the prompt fix, save it as a new agent version, and repeat from the run step.

![A simulation run with the Conversation Details panel open on a failed call, showing the success criteria and the failure reason](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/simulation-results.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=e673132991a926166306e31971df50c1)

## Check the result

When the setup works, a drop in the pass rate traces to one or two metrics that regressed, and you can open the failing calls to see what the agent said at that step. If every metric still passes while the pass rate drops, the failure is in a step that no metric checks yet, so add a metric for it.

Some checks apply only to some calls, for example a callback number only matters when the agent offered a callback. Make that metric conditional, so it runs only when the agent called a given tool or the call reached a given [workflow](/test/simulations/reference.md#workflow) node. On other calls it does not run and has no result, so it does not drag down the comparison. See [Conditional metrics](/test/metrics.md#conditional-metrics).

## Related

* [Metrics](/test/metrics.md)
* [Metrics Lab](/test/metrics.md#metrics-lab)
* [Simulation Results](/test/simulations/create.md#read-the-results)
* [A/B Test Two Agents](/cookbook/ab-test-agents.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.