Metrics

If you are setting up custom metrics for production calls, see Metrics.

The Metrics page with custom metrics grouped by category, each showing its summary and how many agents it runs on

Set up a metric

On the Metrics page, click Create, then Create automatically. Review the drafts and save the ones you want. For a hand-written metric, choose Create manually and use the prompting guide.

The Create Metric dialog with a name and prompt filled in, and the Agents, Evaluates, and Runs on fields set

New simulations run all of their agent’s metrics. For an existing simulation, turn the metric on in its Custom Metrics tab.

Metric types

Type	How it scores
LLM as a Judge	An LLM scores the conversation against your prompt.
Composite	Combines the results of other metrics into one score or label.
Tool Call	Passes when the agent called the tools you name.
Phrase Match	Checks whether specific words were said.
Transfer Landed	Passes when the Digital Human you choose answers the transfer within a time limit counted from the start of the call. The default is 180 seconds and the maximum is 900.

Response types

Choose the response type that matches the question your prompt asks:

Response type	Example
Pass/Fail	Did the agent verify identity before it shared account details?
Yes/No	Did the customer ask for a human?
Number	Rate the clarity of the explanation from 1 to 5.
Text	Summarize the objections the customer raised.
Categorical	Classify the call as `resolved`, `escalated`, or `unresolved`.
Structured JSON	Return (boolean) and `flags` (string array).

Prompting guide

An LLM as a Judge metric sends your prompt and the conversation to the judge, the AI model that scores the metric. A vague prompt can make the judge pass one call and fail an almost identical call.

Rules for all response types

Write criteria that a reader can answer from the transcript or the audio.
Test one thing per metric. If two results matter independently, write two metrics.
Replace words such as “good”, “appropriate”, or “well” with behavior that you can see in the transcript.
Write what the metric verifies, not what the agent must do. Agent instructions go in the agent prompt.

Prompts by response type

Pass/Fail
Number
Text
Categorical
Structured JSON

State the pass condition and the fail condition. Give one example of each.

The metric passes if the agent asked for at least two of: full name,
date of birth, last four digits of the card, or billing ZIP code,
BEFORE it shared any account details.

The metric fails if the agent asked for fewer than two, or shared
account details first.

Example pass: agent asked for date of birth and last four, then
quoted the balance.
Example fail: agent quoted the balance after it confirmed only the name.

Do not write “Did the agent verify and help the customer well?” That prompt tests two behaviors and uses an undefined word.

Define each level on the scale. Use five levels or fewer. With no definitions, the judge can give a 2 on one call and a 4 on an almost identical call.

Rate the clarity of the agent's explanation from 1 to 5.

1 = Confusing. The customer cannot act on the explanation.
2 = Partial. The agent used key terms without a definition.
3 = Adequate. The customer can act, but must guess some steps.
4 = Clear. The agent named the steps in order.
5 = Exceptional. The agent named the steps in order and confirmed
    them with the customer.

Set a fixed structure and a word limit, so that you can compare outputs across calls. Ask for observations, not opinions.

Summarize how the agent handled customer objections.

Include:
1. Each objection the customer raised, in order.
2. The agent's response to each, quoted briefly.
3. Whether the response resolved the objection (yes, partial, no).

Keep the response under 120 words.

Define the evidence for each label. Give one rule for calls that match more than one label. Use seven labels or fewer, and put rare cases in an `other` label.

Classify this call into exactly one category.

- new_order: The customer places an order they did not place before.
- modify_order: The customer changes or adds to an existing order.
- order_status: The customer asks where an existing order is.
- complaint: The customer reports a problem with an order or the service.
- general_inquiry: The customer asks about hours, locations, menu,
  or prices and does not place or change an order.

If two categories apply, select the category that used the most call time.

Give each key a type and a definition, and tell the judge to return only the object. Put the same shape in the metric's JSON Schema field, because the judge output must match it.

Return a JSON object with exactly these keys:

- identity_verified (boolean): true if the agent verified identity with
  at least two factors before it shared account details.
- disclosure_read (boolean): true if the agent read the recording
  disclosure in the first 15 seconds.
- flags (string[]): one short sentence for each compliance issue.
  Empty array if there are none.

Return only the JSON object.

Dynamic variables

To put a call-specific value into a metric, such as the customer’s name, use a dynamic variable. Write {{key}} in the metric’s Prompt. Before the judge runs, Bluejay replaces each {{key}} with the value of key in the conversation metadata.

Prompt:    Did the agent confirm the account ending in {{account_last_four}}
           for {{customer_name}}?
metadata:  { "customer_name": "Sarah", "account_last_four": "4821" }
Judge sees: Did the agent confirm the account ending in 4821 for Sarah?

Each top-level key in the conversation metadata is a variable. If you send conversation data through the API, put the keys in the request’s metadata object, as in the example request. If a voice platform sends your calls to Bluejay, the integration page for that platform says if its call metadata is included. A key can contain only letters, digits, and underscores, and is case-sensitive, so {{customer name}} and {{order.id}} stay as written.

If a key is missing, the judge sees undefined in the prompt and can give a wrong verdict. Make sure each key in the prompt is in the metadata of every conversation that the metric evaluates.

Composite metrics

Use a composite metric to roll several metrics into one score or label. For example, combine a clarity score and a Pass/Fail check into one Scheduling Quality score. A composite metric reads results that the other metrics already produced, so it makes no LLM call. Set Output to one of these:

Numerical: write a formula over the metrics.
Categorical: add an option for each label, with the conditions that select it.

In YAML, see the composite metric example.

Conditional metrics

Use a conditional metric when a rule applies only to some conversations. For example, check the refund policy wording only when the agent called the refund tool. On other conversations, the metric does not run and has no result. A condition can check that:

The conversation reached a node of the agent workflow. You pick the node from the workflow, so the metric must grade one agent. In YAML, find node IDs with Get Workflow Summary.
The agent called a given tool.
Another metric has a given result, for example Fail.

In YAML, see the conditional metric example. A composite or conditional metric can read only metrics that always run.

Tool calls, events, and metadata

A transcript does not show which tools your agent called. To grade tool calls, send them to Bluejay with the conversation. You can send events and metadata in the same way. metadata is a free-form JSON object for the whole conversation, and its keys fill a metric’s dynamic variables.

Send tool calls

Bluejay sends an X-Simulation-Result-Id with each simulated call, and the integration page for your connection says where it arrives. Have your agent read the ID, record its tool calls during the call, and send them with the ID after the call ends. If your agent cannot read the ID, leave out simulation_result_id and send start_time, end_time, and duration so Bluejay can find the call. After the call ends, send the data to POST /v1/update-simulation-result:

{
  "simulation_result_id": "1234",
  "tool_calls": [
    {
      "name": "transfer_funds",
      "parameters": { "from": "12345", "to": "67890", "amount": 500 },
      "output": { "transaction_id": "txn_abc123", "status": "completed" },
      "start_offset_ms": 120000
    }
  ],
  "events": [
    { "title": "Customer verified", "start_offset_ms": 30000, "tags": ["verification"] }
  ],
  "metadata": { "resolution_status": "resolved" }
}

Check your platform’s integration page first: some integrations collect tool calls without this API call.

Tool call fields

Field	Type	Required	Description
`name`	string	Yes	The tool name.
	integer	Yes	Milliseconds from the start of the conversation.
`description`	string	No	What the tool does.
`parameters`	object	No	The input to the tool.
`output`	any	No	The value the tool returned.

Event fields

An event marks a moment in the conversation, for example an escalation or a hold.

Field	Type	Required	Description
`title`	string	Yes	The event name.
	integer	Yes	Milliseconds from the start of the conversation.
	integer	No	When the event ended.
`description`	string	No	Details of the event.
`tags`	string[]	No	Tags for the event.
`metadata`	object	No	Extra data for the event.

Metrics Lab

You label conversations with the verdict you expect, then test prompt revisions until the metric agrees with you.

Build a lab

A lab pairs one or more custom metrics with a sample of conversations from an agent or a simulation run. You test prompt revisions against this sample, so the result is only as good as the sample. Select 20 to 50 conversations for each metric, and include clear passes, clear fails, and unclear cases.

The Lab tab on the Metrics page with lab cards and the Create Lab button

The Create Lab dialog with the name, metrics, and conversations steps

Label the conversations

Your label is the human verdict, and the metric’s result is the AI verdict. The alignment score shows how often the two agree. To save time, label in bulk when many rows share the same verdict.

The Annotate tab with AI and Human columns

Label every conversation before you read the alignment score. The score compares only rows that have a human verdict.

To test prompt revisions and publish the best one, see Improve the accuracy of a metric.

Get Started

Test

Monitor

Improve

Cookbook

Integrations

Administration

Set up a metric

Metric types

Response types

Prompting guide

Rules for all response types

Prompts by response type

Dynamic variables

Composite metrics

Conditional metrics

Tool calls, events, and metadata

Send tool calls

Tool call fields

Event fields

Metrics Lab

Build a lab

Label the conversations

​Set up a metric

​Metric types

​Response types

​Prompting guide

​Rules for all response types

​Prompts by response type

​Dynamic variables

​Composite metrics

​Conditional metrics

​Tool calls, events, and metadata

​Send tool calls

​Tool call fields

​Event fields

​Metrics Lab

​Build a lab

​Label the conversations

​Related

Set up a metric

Metric types

Response types

Prompting guide

Rules for all response types

Prompts by response type

Dynamic variables

Composite metrics

Conditional metrics

Tool calls, events, and metadata

Send tool calls

Tool call fields

Event fields

Metrics Lab

Build a lab

Label the conversations

Related