Best practices for writing Custom Metric prompts that produce reliable, predictable evaluations
A Custom Metric is a verifier. It either confirms that an agent or customer took an action on the call, or it rates, ranks, or classifies what they did. Because the verdict drives what your team trusts, the prompt that defines it has to be written clearly.The biggest payoff of a well-written metric is predictable evaluations. When the prompt is precise, the verdict matches what your team would have said reading the transcript or listening to the audio. When the prompt is loose, the model has no solid definition of success and the verdict becomes non-deterministic. The same call returns different scores on different runs.
A good metric returns the verdict you would have given yourself reviewing the call. If your gut and the metric disagree, the prompt is the problem, not the model. Tighten the wording or add the missing piece of evidence.
Each response type does a different job, so each one rewards a different prompting style. Pick the tab that matches the metric you are writing.
Pass / Fail
Quantitative
Qualitative
Enum
JSON
A Pass / Fail metric is an assertion. State the conditions that make the metric pass and the conditions that make it fail.Good prompt
The metric passes if the agent verified the customer's identity byasking for at least two of: full name, date of birth, last fourdigits of the card, or billing ZIP code, BEFORE sharing any accountdetails.The metric fails if fewer than two were requested, or if accountdetails were shared first.Example pass: agent asked for date of birth and last four, thenquoted the balance.Example fail: agent quoted the balance after only confirming thecustomer's name.
Bad prompt
Did the agent verify and help the customer well during the call?
The good version names the pass condition, the fail condition, and gives one example of each. The bad version stacks two behaviors and leans on an undefined word.Common Pass / Fail mistakesStacking independent behaviors. A Pass / Fail metric should test one thing you want to know. If verifying identity and issuing a refund are two outcomes you would care about independently, write two metrics. If they only matter as a joint result (the call is a fail unless both happened), one metric is fine.Vague pass criteria. “The agent did the right thing” leaves the definition to the model. Replace it with the specific behavior you would look for in the transcript.Describing what the agent should do, not what the metric verifies. The prompt grades the call. Phrases like “the agent should be polite” belong in the agent’s instructions, not in a metric.
If the prompt reaches for words like “good,” “appropriate,” or “well,” you are leaving the definition to the model. Replace those words with specific, observable evidence.
Dynamic variablesUse dynamic variables to make the pass criteria call-specific.
The metric passes if the agent confirmed the order for {{customer_name}}matched the items in {{order_summary}}.
A quantitative metric is a rubric. Define every level on the scale so the judge has anchors to grade against.Good prompt
Rate the agent's clarity of explanation from 1 to 5.1 = Confusing. The customer would not be able to act on the explanation.2 = Partial. Key terms were used without definition.3 = Adequate. The customer could likely act, but had to infer some steps.4 = Clear. Steps were named and ordered.5 = Exceptional. Steps were named, ordered, and confirmed back with the customer.
Bad prompt
Rate the clarity from 1 to 10 where 1 is bad and 10 is good.
A 1-to-10 scale with no anchors returns a 4 on one call and an 8 on a near-identical one. A 1-to-5 scale with anchored levels stays stable.Common quantitative mistakesUnanchored scales. Without a definition for each value, the model invents its own scale and your scores drift across runs.Too many levels. Models reliably distinguish about five levels. Ten or a hundred sound precise but produce more noise than signal.Vague endpoints. “1 is bad, 5 is good” tells the judge nothing. Anchor 1 and 5 to observable behaviors so the middle values land in predictable places.Dynamic variablesInject call-specific anchors when the rubric depends on facts that change per call.
Rate how thoroughly the agent explained the {{plan_name}} plan from 1 to 5.
A qualitative metric returns free text. The trick is to constrain the structure so the outputs stay comparable across calls.Good prompt
Summarize how the agent handled customer objections.Include:1. The objections the customer raised, in the order they were raised.2. The agent's response to each, quoted briefly.3. Whether the response resolved the objection (yes, partial, no).Keep the response under 120 words.
Bad prompt
Tell me how the call went. Be detailed.
Free-text answers are useful only when their structure is fixed. Otherwise comparing them across calls is impossible.Common qualitative mistakesNo structure. “Tell me about the call” produces a different summary every run. Numbered sections or required fields fix the shape.No length cap. Without a word limit, summaries balloon and stop being skimmable. 100 to 150 words is usually plenty.Asking for opinions instead of observations. “How do you feel about the agent’s tone” is unfalsifiable. “Quote the phrase the agent used to acknowledge the customer’s frustration” is observable.Dynamic variablesInject the topic the summary should focus on.
Summarize how the agent handled the customer's question about {{topic}}.
An enum metric is a multi-class classifier. Define the attributes that put a call into one class versus another. The judge needs to know what evidence makes a call a member of each class.Good prompt
Classify this fast-food call into exactly one category.- new_order: The customer is placing an order they have not placed before.- modify_order: The customer is changing or adding to an order already placed.- order_status: The customer is asking where their existing order is or when it will be ready.- complaint: The customer is reporting a problem with an order, the food, or the service.- general_inquiry: The customer is asking about hours, locations, menu items, or pricing without placing or modifying an order.If two categories apply (for example, a complaint that turns into a neworder), pick the category that occupied the most call time.
Bad prompt
What kind of call is this? new_order, modify_order, order_status,complaint, or general_inquiry?
The bad version names the labels but defines none of them. Calls land in different buckets across runs.Common enum mistakesLabels without definitions. “Resolved, escalated, callback_scheduled” looks self-explanatory and is not. Spell out the evidence that puts a call in each bucket.No tiebreaker. Real calls overlap. State a single rule the judge follows when more than one label could apply.Too many classes. Beyond seven or eight categories, the model starts to blend boundaries. Collapse rare labels into an “other” bucket if you need to.Dynamic variablesInject classification context when the right label depends on facts the judge cannot infer from the transcript alone.
Classify the call. The customer's stated reason for calling was: {{stated_reason}}.
A JSON metric is a typed contract. The schema you define is what the model has to return. Specify keys, types, and what counts as evidence for each.Good prompt
Evaluate the call and return a JSON object with exactly these keys:- identity_verified (boolean): true if the agent verified identity using at least two factors before sharing account details.- disclosure_read (boolean): true if the recording disclosure was read within the first 15 seconds of the call.- consent_obtained (boolean): true if the agent received explicit verbal consent before any account change.- flags (string[]): a short string for each compliance issue you found. Empty array if there were none.Return only the JSON object. Do not include any other text.
Bad prompt
Return JSON about whether the call was compliant.
Specify the keys, the types, and what counts as evidence for each one. The “return only the JSON object” line is what stops the model from adding a friendly preamble.Common JSON mistakesKeys without types. A field called verified could be a boolean, a string, or an enum. State the type so the schema stays stable across runs.No “return only the JSON” instruction. Without it, the model often adds a friendly sentence before or after the object and your downstream parser breaks.Fuzzy field descriptions. “Issues found” is vague. “Each issue as a short sentence describing what was missed and where in the call” tells the judge what to put in the field.Dynamic variablesInject required context into a key’s definition.
verify_id (boolean): true if the agent verified the customer using atleast these factors: {{required_factors}}.
Dynamic variables let you inject call-specific values into a metric prompt at evaluation time. You write {{customer_name}} or {{plan_price}} in the prompt, pass the matching keys in the metadata object on the evaluate request, and Bluejay substitutes them before the judge runs.This pattern matters when the right answer depends on facts that change per call: the customer’s name, the plan they are on, the order ID, the expected balance. Without dynamic variables you would need a separate metric for every value. With them, you keep one metric and pass the values per call.
Dynamic variables work in any response type. The substitution happens before the judge sees the prompt, so anything you can write as plain text can be parameterized.
For the full setup and the request-payload format, see Dynamic Variables.
Sometimes a metric is close but not quite right. The pass rate is too generous, or it flags borderline cases your team would have let through. You have two paths.
Edit the prompt by hand
Add a clarifying sentence, run it against a few sample calls, and check whether the new outputs match your judgment. Fast for small adjustments.
Tune with human feedback
For metrics that need to encode your team’s specific standards, use the Metrics Lab. You annotate calls, the Lab tunes the metric to match your annotations.
The second path is an application of Reinforcement Learning from Human Feedback (RLHF), the technique behind most modern AI assistants. A person rates the model’s output, and the system updates so future outputs match what the person preferred. The Metrics Lab applies that idea to a single Custom Metric.
