LLM as a Judge

Use an LLM as a Judge metric to grade a conversation on a rule that you write in plain language.

Response types

Choose the response type that matches the question your prompt asks:

Response type	Example
Pass/Fail	Did the agent verify identity before it shared account details?
Yes/No	Did the customer ask for a human?
Number	Rate the clarity of the explanation from 1 to 5.
Text	Summarize the objections the customer raised.
Categorical	Classify the call as `resolved`, `escalated`, or `unresolved`.
Structured JSON	Return (boolean) and `flags` (string array).

Prompting guide

An LLM as a Judge metric sends your prompt and the conversation to the judge, the AI model that scores the metric. A vague prompt can make the judge pass one call and fail an almost identical call.

Rules for all response types

Write criteria that a reader can answer from the transcript or the audio.
Test one thing per metric. If two results matter independently, write two metrics.
Replace words such as “good”, “appropriate”, or “well” with behavior that you can see in the transcript.
Write what the metric verifies, not what the agent must do. Agent instructions go in the agent prompt.

Prompts by response type

Pass/Fail
Number
Text
Categorical
Structured JSON

State the pass condition and the fail condition. Give one example of each.

The metric passes if the agent asked for at least two of: full name,
date of birth, last four digits of the card, or billing ZIP code,
BEFORE it shared any account details.

The metric fails if the agent asked for fewer than two, or shared
account details first.

Example pass: agent asked for date of birth and last four, then
quoted the balance.
Example fail: agent quoted the balance after it confirmed only the name.

Do not write “Did the agent verify and help the customer well?” That prompt tests two behaviors and uses an undefined word.

Define each level on the scale. Use five levels or fewer. With no definitions, the judge can give a 2 on one call and a 4 on an almost identical call.

Rate the clarity of the agent's explanation from 1 to 5.

1 = Confusing. The customer cannot act on the explanation.
2 = Partial. The agent used key terms without a definition.
3 = Adequate. The customer can act, but must guess some steps.
4 = Clear. The agent named the steps in order.
5 = Exceptional. The agent named the steps in order and confirmed
    them with the customer.

Set a fixed structure and a word limit, so that you can compare outputs across calls. Ask for observations, not opinions.

Summarize how the agent handled customer objections.

Include:
1. Each objection the customer raised, in order.
2. The agent's response to each, quoted briefly.
3. Whether the response resolved the objection (yes, partial, no).

Keep the response under 120 words.

Define the evidence for each label. Give one rule for calls that match more than one label. Use seven labels or fewer, and put rare cases in an `other` label.

Classify this call into exactly one category.

- new_order: The customer places an order they did not place before.
- modify_order: The customer changes or adds to an existing order.
- order_status: The customer asks where an existing order is.
- complaint: The customer reports a problem with an order or the service.
- general_inquiry: The customer asks about hours, locations, menu,
  or prices and does not place or change an order.

If two categories apply, select the category that used the most call time.

Give each key a type and a definition, and tell the judge to return only the object. Put the same shape in the metric's JSON Schema field, because the judge output must match it.

Return a JSON object with exactly these keys:

- identity_verified (boolean): true if the agent verified identity with
  at least two factors before it shared account details.
- disclosure_read (boolean): true if the agent read the recording
  disclosure in the first 15 seconds.
- flags (string[]): one short sentence for each compliance issue.
  Empty array if there are none.

Return only the JSON object.

Dynamic variables

To put a call-specific value into a metric, such as the customer’s name, use a dynamic variable. Write {{key}} in the metric’s Prompt. Before the judge runs, Bluejay replaces each {{key}} with the value of key in the conversation metadata.

Prompt:    Did the agent confirm the account ending in {{account_last_four}}
           for {{customer_name}}?
metadata:  { "customer_name": "Sarah", "account_last_four": "4821" }
Judge sees: Did the agent confirm the account ending in 4821 for Sarah?

Each top-level key in the conversation metadata is a variable. If you send conversation data through the API, put the keys in the request’s metadata object, as in the example request on the Create a Metric page. If a voice platform sends your calls to Bluejay, the integration page for that platform says if its call metadata is included. A key can contain only letters, digits, and underscores, and is case-sensitive, so {{customer name}} and {{order.id}} stay as written.

If a key is missing, the judge sees undefined in the prompt and can give a wrong verdict. Make sure each key in the prompt is in the metadata of every conversation that the metric evaluates.

Get Started

Test

Monitor

Improve

Cookbook

Integrations

Administration

Response types

Prompting guide

Rules for all response types

Prompts by response type

Dynamic variables

​Response types

​Prompting guide

​Rules for all response types

​Prompts by response type

​Dynamic variables

Response types

Prompting guide

Rules for all response types

Prompts by response type

Dynamic variables