Skip to main content
If you are setting up custom metrics for simulation results, see Metrics.

Set up a metric

1

Open the form

On the Metrics page, click Create, then Create manually.The Metrics page with the Create menu open, showing Add from library, Create automatically, and Create manuallyThe Metrics page with the Create menu open, showing Add from library, Create automatically, and Create manually
2

Write the rule

Choose a metric type. For LLM as a Judge, pick the response type and write the prompt with the prompting guide.The Create Metric dialog with a name and prompt filled in, and the Agents, Evaluates, and Runs on fields setThe Create Metric dialog with a name and prompt filled in, and the Agents, Evaluates, and Runs on fields set
3

Choose where it runs

Select the agents to grade. Runs on decides whether the metric grades simulations, production conversations, or both. To grade only some conversations, add metric conditions. Then click Create.

Metric types

Predicted WER

Predicted WER estimates how many words your agent’s transcript got wrong. When you send a production conversation with both recording_url and a transcript, Bluejay transcribes the recording itself and compares the two. A value of 0.12 means about 12% of words differ. To turn it on, add Predicted WER from the metrics library, then turn on Enable Predicted Word Error Rate Evaluations under Settings > Observability. Each conversation is transcribed one extra time, which counts toward your speech-to-text usage. Conversations sent with only a transcript, or only a recording, get no value. Read it in the metrics array of GET /v1/retrieve-call-log/ and of the observability_call_evaluated webhook, as the entry with metric_name set to predicted_wer.

Add metric conditions

Use a conditional metric when a rule applies only to some conversations. On other conversations, the metric does not run and has no result. Common use cases include:
  • Check the refund policy wording only when the agent called the refund tool.
  • Check that the agent stated the copay only when the conversation reached the billing step of the workflow.
  • Check how the agent handled the escalation only when a Customer Asked for a Human metric returned Yes.
A condition can check that:
  • The conversation reached a node of the agent workflow. You pick the node from the workflow, so the metric must grade one agent. In YAML, find node IDs with Get Workflow Summary.
  • The agent called a given tool.
  • Another metric has a given result, for example Fail.
A condition can read only metrics that always run. In YAML, see the conditional metric example.

Tool calls, events, and metadata

If a metric grades something the transcript does not show, such as which tools your agent called, send that data with each production conversation. You can send:
  • Tool calls: the tools your agent called, with their inputs and outputs. A Tool Call metric needs them.
  • Traces: the ID of the OpenTelemetry trace your agent exported during the call, sent as trace_id. A Trace metric needs it, and every metric sees a summary of the linked trace. See Traces.
  • Events: moments in the conversation, such as an escalation or a hold.
  • Metadata: a JSON object about the whole conversation, such as the customer’s name. Each key fills a dynamic variable in a metric prompt.

Send tool calls

Put tool_calls, events, and metadata in the same POST /v1/evaluate request as the transcript or recording, because you cannot add them later.

Tool call fields

Event fields

An event marks a moment in the conversation, for example an escalation or a hold.
Last modified on October 31, 2026