> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM as a Judge

> Grade conversations with a prompt.

Use an LLM as a Judge metric to grade a conversation on a rule that you write in plain language.

## Response types

Choose the response type that matches the question your prompt asks:

| Response type | Example |
| - | - |
| Pass/Fail | Did the agent verify identity before it shared account details? |
| Yes/No | Did the customer ask for a human? |
| Number | Rate the clarity of the explanation from 1 to 5. |
| Text | Summarize the objections the customer raised. |
| Categorical | Classify the call as `resolved`, `escalated`, or `unresolved`. |
| Structured JSON | Return identity\_verified (boolean) and `flags` (string array). |

## Prompting guide

An LLM as a Judge metric sends your prompt and the conversation to the judge, the AI model that scores the metric. A vague prompt can make the judge pass one call and fail an almost identical call.

### Rules for all response types

* Write criteria that a reader can answer from the transcript or the audio.
* Test one thing per metric. If two results matter independently, write two metrics.
* Replace words such as "good", "appropriate", or "well" with behavior that you can see in the transcript.
* Write what the metric verifies, not what the agent must do. Agent instructions go in the agent prompt.

### Prompts by response type

**Pass/Fail**

```
State the pass condition and the fail condition. Give one example of each.

The metric passes if the agent asked for at least two of: full name,
date of birth, last four digits of the card, or billing ZIP code,
BEFORE it shared any account details.

The metric fails if the agent asked for fewer than two, or shared
account details first.

Example pass: agent asked for date of birth and last four, then
quoted the balance.
Example fail: agent quoted the balance after it confirmed only the name.
```

Do not write "Did the agent verify and help the customer well?" That prompt tests two behaviors and uses an undefined word.

**Number**

```
Define each level on the scale. Use five levels or fewer. With no definitions, the judge can give a 2 on one call and a 4 on an almost identical call.

Rate the clarity of the agent's explanation from 1 to 5.

1 = Confusing. The customer cannot act on the explanation.
2 = Partial. The agent used key terms without a definition.
3 = Adequate. The customer can act, but must guess some steps.
4 = Clear. The agent named the steps in order.
5 = Exceptional. The agent named the steps in order and confirmed
    them with the customer.
```

**Text**

```
Set a fixed structure and a word limit, so that you can compare outputs across calls. Ask for observations, not opinions.

Summarize how the agent handled customer objections.

Include:
1. Each objection the customer raised, in order.
2. The agent's response to each, quoted briefly.
3. Whether the response resolved the objection (yes, partial, no).

Keep the response under 120 words.
```

**Categorical**

```
Define the evidence for each label. Give one rule for calls that match more than one label. Use seven labels or fewer, and put rare cases in an `other` label.

Classify this call into exactly one category.

- new_order: The customer places an order they did not place before.
- modify_order: The customer changes or adds to an existing order.
- order_status: The customer asks where an existing order is.
- complaint: The customer reports a problem with an order or the service.
- general_inquiry: The customer asks about hours, locations, menu,
  or prices and does not place or change an order.

If two categories apply, select the category that used the most call time.
```

**Structured JSON**

```
Give each key a type and a definition, and tell the judge to return only the object. Put the same shape in the metric's JSON Schema field, because the judge output must match it.

Return a JSON object with exactly these keys:

- identity_verified (boolean): true if the agent verified identity with
  at least two factors before it shared account details.
- disclosure_read (boolean): true if the agent read the recording
  disclosure in the first 15 seconds.
- flags (string[]): one short sentence for each compliance issue.
  Empty array if there are none.

Return only the JSON object.
```

## Dynamic variables

To put a call-specific value into a metric, such as the customer's name, use a dynamic variable. Write `{{key}}` in the metric's **Prompt**. Before the judge runs, Bluejay replaces each `{{key}}` with the value of `key` in the conversation `metadata`.

```
Prompt:    Did the agent confirm the account ending in {{account_last_four}}
           for {{customer_name}}?
metadata:  { "customer_name": "Sarah", "account_last_four": "4821" }
Judge sees: Did the agent confirm the account ending in 4821 for Sarah?
```

Each top-level key in the conversation `metadata` is a variable. If you send conversation data through the API, put the keys in the request's `metadata` object, as in the example request on the Create a Metric page. If a voice platform sends your calls to Bluejay, the integration page for that platform says if its call metadata is included.

A key can contain only letters, digits, and underscores, and is case-sensitive, so `{{customer name}}` and `{{order.id}}` stay as written.

> [!WARNING]
>
> If a key is missing, the judge sees `undefined` in the prompt and can give a wrong verdict. Make sure each key in the prompt is in the `metadata` of every conversation that the metric evaluates.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.