> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Metrics

> Grade conversations on your own rules.

If you are setting up custom metrics for production calls, see [Metrics](/monitor/metrics.md).

![The Metrics page with custom metrics grouped by category, each showing its summary and how many agents it runs on](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-page.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=49f6629dcd5d7eedf1dd75d85d8204b2)

<ul aria-label="Example metric results" />

## Set up a metric

On the **Metrics** page, click **Create**, then **Create automatically**. Review the drafts and save the ones you want.

For a hand-written metric, choose **Create manually** and use the [prompting guide](#prompting-guide).

![The Create Metric dialog with a name and prompt filled in, and the Agents, Evaluates, and Runs on fields set](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metric-create-dialog.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=859613c65313758fa8501964443f47e5)

New simulations run all of their agent's metrics. For an existing simulation, turn the metric on in its **Custom Metrics** tab.

## Metric types

| Type | How it scores |
| - | - |
| LLM as a Judge | An LLM scores the conversation against your prompt. |
| Composite | Combines the results of other metrics into one score or label. |
| Tool Call | Passes when the agent called the tools you name. |
| Phrase Match | Checks whether specific words were said. |
| Transfer Landed | Passes when the Digital Human you choose answers the transfer within a time limit counted from the start of the call. The default is 180 seconds and the maximum is 900. |

## Response types

Choose the response type that matches the question your prompt asks:

| Response type | Example |
| - | - |
| Pass/Fail | Did the agent verify identity before it shared account details? |
| Yes/No | Did the customer ask for a human? |
| Number | Rate the clarity of the explanation from 1 to 5. |
| Text | Summarize the objections the customer raised. |
| Categorical | Classify the call as `resolved`, `escalated`, or `unresolved`. |
| Structured JSON | Return identity\_verified (boolean) and `flags` (string array). |

## Prompting guide

An LLM as a Judge metric sends your prompt and the conversation to the judge, the AI model that scores the metric. A vague prompt can make the judge pass one call and fail an almost identical call.

### Rules for all response types

* Write criteria that a reader can answer from the transcript or the audio.
* Test one thing per metric. If two results matter independently, write two metrics.
* Replace words such as "good", "appropriate", or "well" with behavior that you can see in the transcript.
* Write what the metric verifies, not what the agent must do. Agent instructions go in the agent prompt.

### Prompts by response type

**Pass/Fail**

```
State the pass condition and the fail condition. Give one example of each.

The metric passes if the agent asked for at least two of: full name,
date of birth, last four digits of the card, or billing ZIP code,
BEFORE it shared any account details.

The metric fails if the agent asked for fewer than two, or shared
account details first.

Example pass: agent asked for date of birth and last four, then
quoted the balance.
Example fail: agent quoted the balance after it confirmed only the name.
```

Do not write "Did the agent verify and help the customer well?" That prompt tests two behaviors and uses an undefined word.

**Number**

```
Define each level on the scale. Use five levels or fewer. With no definitions, the judge can give a 2 on one call and a 4 on an almost identical call.

Rate the clarity of the agent's explanation from 1 to 5.

1 = Confusing. The customer cannot act on the explanation.
2 = Partial. The agent used key terms without a definition.
3 = Adequate. The customer can act, but must guess some steps.
4 = Clear. The agent named the steps in order.
5 = Exceptional. The agent named the steps in order and confirmed
    them with the customer.
```

**Text**

```
Set a fixed structure and a word limit, so that you can compare outputs across calls. Ask for observations, not opinions.

Summarize how the agent handled customer objections.

Include:
1. Each objection the customer raised, in order.
2. The agent's response to each, quoted briefly.
3. Whether the response resolved the objection (yes, partial, no).

Keep the response under 120 words.
```

**Categorical**

```
Define the evidence for each label. Give one rule for calls that match more than one label. Use seven labels or fewer, and put rare cases in an `other` label.

Classify this call into exactly one category.

- new_order: The customer places an order they did not place before.
- modify_order: The customer changes or adds to an existing order.
- order_status: The customer asks where an existing order is.
- complaint: The customer reports a problem with an order or the service.
- general_inquiry: The customer asks about hours, locations, menu,
  or prices and does not place or change an order.

If two categories apply, select the category that used the most call time.
```

**Structured JSON**

```
Give each key a type and a definition, and tell the judge to return only the object. Put the same shape in the metric's JSON Schema field, because the judge output must match it.

Return a JSON object with exactly these keys:

- identity_verified (boolean): true if the agent verified identity with
  at least two factors before it shared account details.
- disclosure_read (boolean): true if the agent read the recording
  disclosure in the first 15 seconds.
- flags (string[]): one short sentence for each compliance issue.
  Empty array if there are none.

Return only the JSON object.
```

### Dynamic variables

To put a call-specific value into a metric, such as the customer's name, use a dynamic variable. Write `{{key}}` in the metric's **Prompt**. Before the judge runs, Bluejay replaces each `{{key}}` with the value of `key` in the conversation `metadata`.

```
Prompt:    Did the agent confirm the account ending in {{account_last_four}}
           for {{customer_name}}?
metadata:  { "customer_name": "Sarah", "account_last_four": "4821" }
Judge sees: Did the agent confirm the account ending in 4821 for Sarah?
```

Each top-level key in the conversation `metadata` is a variable. If you send conversation data through the API, put the keys in the request's `metadata` object, as in the [example request](#send-tool-calls). If a voice platform sends your calls to Bluejay, the integration page for that platform says if its call metadata is included.

A key can contain only letters, digits, and underscores, and is case-sensitive, so `{{customer name}}` and `{{order.id}}` stay as written.

> [!WARNING]
>
> If a key is missing, the judge sees `undefined` in the prompt and can give a wrong verdict. Make sure each key in the prompt is in the `metadata` of every conversation that the metric evaluates.

## Composite metrics

Use a composite metric to roll several metrics into one score or label. For example, combine a clarity score and a Pass/Fail check into one Scheduling Quality score. A composite metric reads results that the other metrics already produced, so it makes no LLM call.

Set **Output** to one of these:

* **Numerical**: write a formula over the metrics.
* **Categorical**: add an option for each label, with the conditions that select it.

In YAML, see the [composite metric example](/bluejay-as-code/cookbook.md#composite-metrics).

## Conditional metrics

Use a conditional metric when a rule applies only to some conversations. For example, check the refund policy wording only when the agent called the refund tool. On other conversations, the metric does not run and has no result. A condition can check that:

* The conversation reached a node of the agent [workflow](/test/simulations/reference.md#workflow). You pick the node from the workflow, so the metric must grade one agent. In YAML, find node IDs with [Get Workflow Summary](/api-reference/endpoint/get-workflow-summary.md).
* The agent called a given tool.
* Another metric has a given result, for example Fail.

In YAML, see the [conditional metric example](/bluejay-as-code/cookbook.md#conditional-metrics).

A composite or conditional metric can read only metrics that always run.

## Tool calls, events, and metadata

A transcript does not show which tools your agent called. To grade tool calls, send them to Bluejay with the conversation. You can send events and metadata in the same way. `metadata` is a free-form JSON object for the whole conversation, and its keys fill a metric's [dynamic variables](#dynamic-variables).

### Send tool calls

Bluejay sends an `X-Simulation-Result-Id` with each simulated call, and the [integration page](/integrations/phone-number.md) for your connection says where it arrives. Have your agent read the ID, record its tool calls during the call, and send them with the ID after the call ends.

If your agent cannot read the ID, leave out `simulation_result_id` and send `start_time`, `end_time`, and `duration` so Bluejay can find the call.

After the call ends, send the data to [`POST /v1/update-simulation-result`](/api-reference/endpoint/update-simulation-result):

```json theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
{
  "simulation_result_id": "1234",
  "tool_calls": [
    {
      "name": "transfer_funds",
      "parameters": { "from": "12345", "to": "67890", "amount": 500 },
      "output": { "transaction_id": "txn_abc123", "status": "completed" },
      "start_offset_ms": 120000
    }
  ],
  "events": [
    { "title": "Customer verified", "start_offset_ms": 30000, "tags": ["verification"] }
  ],
  "metadata": { "resolution_status": "resolved" }
}
```

Check your platform's integration page first: some integrations collect tool calls without this API call.

### Tool call fields

| Field | Type | Required | Description |
| - | - | - | - |
| `name` | string | Yes | The tool name. |
| start\_offset\_ms | integer | Yes | Milliseconds from the start of the conversation. |
| `description` | string | No | What the tool does. |
| `parameters` | object | No | The input to the tool. |
| `output` | any | No | The value the tool returned. |

### Event fields

An event marks a moment in the conversation, for example an escalation or a hold.

| Field | Type | Required | Description |
| - | - | - | - |
| `title` | string | Yes | The event name. |
| start\_offset\_ms | integer | Yes | Milliseconds from the start of the conversation. |
| end\_offset\_ms | integer | No | When the event ended. |
| `description` | string | No | Details of the event. |
| `tags` | string\[] | No | Tags for the event. |
| `metadata` | object | No | Extra data for the event. |

## Metrics Lab

You label conversations with the verdict you expect, then test prompt revisions until the metric agrees with you.

[https://www.youtube.com/embed/9OaFvPKCvaI](https://www.youtube.com/embed/9OaFvPKCvaI)

### Build a lab

A lab pairs one or more custom metrics with a sample of conversations from an agent or a simulation run. You test prompt revisions against this sample, so the result is only as good as the sample. Select 20 to 50 conversations for each metric, and include clear passes, clear fails, and unclear cases.

![The Lab tab on the Metrics page with lab cards and the Create Lab button](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-landing.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=13d692dc70c449b65fedc5dc9f4e6c21)

![The Create Lab dialog with the name, metrics, and conversations steps](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-create-dialog.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=202e1fafd7062300e4b66fab13ed0ef3)

### Label the conversations

Your label is the human verdict, and the metric's result is the AI verdict. The alignment score shows how often the two agree. To save time, label in bulk when many rows share the same verdict.

![The Annotate tab with AI and Human columns](https://mintcdn.com/bluejay/eKcFBkjZX47pI8Ss/images/metrics-lab-annotate-table.png?fit=max\&auto=format\&n=eKcFBkjZX47pI8Ss\&q=85\&s=35d085aea530a0f9bc7aaddb6324a91a)

> [!NOTE]
>
> Label every conversation before you read the alignment score. The score compares only rows that have a human verdict.

To test prompt revisions and publish the best one, see [Improve the accuracy of a metric](/cookbook/improve-metric-accuracy.md).

## Related

* [Custom Metrics API](/api-reference/endpoint/create-custom-metrics.md)
* [Custom Metrics MCP tools](/mcp/testing-tools.md#metrics)
* [Evaluate API](/api-reference/endpoint/evaluate.md)
* [Metrics Lab API](/api-reference/endpoint/create-lab.md)
* [Metrics Lab MCP tools](/mcp/testing-tools.md#labs)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.