> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Improve the Accuracy of a Metric

> Make a custom metric agree with your verdicts.

Use a Metrics Lab to test prompt revisions of a custom metric against conversations you have graded, and publish the one that agrees with you most.

<video controls muted playsInline preload="metadata" src="https://mintcdn.com/bluejay/dF8I5qGAanSDLxfr/videos/cookbook-metric-accuracy-walkthrough.mp4?fit=max&auto=format&n=dF8I5qGAanSDLxfr&q=85&s=d36c9540cc94ba258f5126a2b6949d0f" />

## Before you start

* Create the metric as an LLM as a Judge metric with a Pass/Fail response type. See [Metrics](/test/metrics.md) and the [Prompting Guide](/test/metrics.md#prompting-guide).
* Have at least 20 scored conversations (from simulated or production calls).

## Steps

**1. Create a lab**

Open **Metrics**, go to the **Lab** tab, and click **Create Lab**. Name the lab, select the metric to tune, then select conversations from **Logs** (production) or **Simulations**.

Pick 20 to 50 conversations that include clear passes, clear fails, and the unclear cases the metric gets wrong. The result is only as good as this sample. See [Build a lab](/test/metrics.md#build-a-lab).

![The Lab tab on the Metrics page with lab cards and the Create Lab button](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-landing.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=13d692dc70c449b65fedc5dc9f4e6c21)

![The Create Lab dialog with the name, metrics, and conversations steps](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-create-dialog.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=202e1fafd7062300e4b66fab13ed0ef3)

**2. Mark the correct answer for each conversation**

On the **Annotate** tab, set the **Human** verdict to what the metric should have returned. Where a call is borderline, click the Human cell and write your reasoning, so you remember why later.

When many rows share a verdict, select them and apply one value, with optional reasoning, to all of them.

Label every conversation before you read the alignment score. It compares only rows that have a **Human** verdict.

![The Annotate tab with AI and Human columns, two rows labeled Pass and one not yet labeled](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-annotate-table.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=35d085aea530a0f9bc7aaddb6324a91a)

![The Bulk Annotate dialog with Role Appropriateness as the metric, Pass as the value, and a reasoning note filled in](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-metric-accuracy-bulk-annotate.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=fe16d6c4e701f8499d99dfd2401fca60)

**3. Find where the metric disagrees**

On the **Annotate** rows, compare the **AI reasoning** with yours wherever the **AI** verdict differs from the **Human** verdict. The difference usually points to a gap in the prompt. In the example above, you passed a call where the agent confirmed the time in its closing message, so the prompt needs to say that counts as repeating the time back.

Then open the **Optimize** tab. The **Alignment** badge on **Published** is your baseline score. Green is 80% or higher, yellow is 50 to 79%, and red is below 50%.

![The Optimize tab with Published selected, an Alignment badge of 50%, and the metric prompt below](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-metric-accuracy-optimize-baseline.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=2f19e6ca6ec51e36531aebb4cda64ec7)

**4. Create and test a new revision**

Click **Improve** above the prompt to get a rewrite based on the disagreements. Read the highlighted changes, then click **Apply** to keep them or **Discard** to drop them. You can also edit the prompt yourself.

Click **Run**. Bluejay saves the prompt as a new revision and scores every conversation in the lab with it. Use the arrows next to the revision name to compare scores between revisions. Repeat until the alignment stops improving.

![The Optimize tab with an Improve rewrite highlighted in the prompt and the Apply and Discard buttons below it](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-optimize-table.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=611149e4d89d0e31905c8961dfd3cfaf)

![The Optimize tab with Revision 2 selected, an Alignment badge of 53%, and the blue Publish button](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-metric-accuracy-run-revision.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=f447908e6ad25e00c55e73239c787c57)

**5. Publish the best revision**

Select the revision that agrees with you most and click **Publish**. This updates the custom metric itself, so every simulation and production evaluation uses the new prompt from then on.

![The Optimize tab after publishing, with Revision 1 selected, an Alignment badge of 100%, and the Publish button greyed out](https://mintcdn.com/bluejay/UqiH6o220FskMm0H/images/cookbook-metric-accuracy-published.png?fit=max\&auto=format\&n=UqiH6o220FskMm0H\&q=85\&s=5faca21b5ff145ab473ba5907d81bfdd)

If you edited the prompt and did not click **Run**, **Publish** still saves it as a revision and updates the metric, but it does not score the lab, so you will not see an alignment score for that prompt.

## Check the result

If the alignment stays low after several revisions, the sample may be too small or one-sided. Add conversations that cover the cases the metric gets wrong with **Add Conversations**, label them, and run again.

For a metric that returns a number, turn on fuzzy alignment in the lab **Settings** so scores within the threshold count as agreeing.

## Related

* [Metrics Lab](/test/metrics.md#metrics-lab)
* [Metrics](/test/metrics.md)
* [Custom Metrics Prompting Guide](/test/metrics.md#prompting-guide)
* [Find Which Check a Regression Broke](/cookbook/find-regression-step.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.