Skip to main content

Agent

The voice, text, or email AI system that you want Bluejay to test and monitor. Your simulations, conversations, and custom metrics all belong to an agent.

Agent Variables

Values that every Digital Human of an agent knows, for example a business name. A Digital Human’s trait with the same name overrides one.

Agent Version

A saved, numbered copy of an agent prompt. Labels such as production and latest point to a version and decide which one a run uses.

Alert

A rule that notifies your team when a metric on production conversations crosses a threshold.

Binary Frame

A WebSocket message that carries raw bytes instead of text. In the CHIRP protocol, binary frames carry the audio.

Bluejay AI

The assistant inside the Bluejay app. It tells you why calls failed and can make the change for you, for example create a simulation, a custom metric, or a dashboard.

Bluejay as Code

Keeping your agents, simulations, Digital Humans, and custom metrics in a YAML file. You pull the file from Bluejay, edit it, and push it back, by hand or from CI.

Call Status

Where a simulated call is in its life, for example Queued, Running, or Completed. A call that fails also has an error code that says why.

CHIRP

The protocol that Bluejay uses to exchange audio and text events with your server over a WebSocket.

Community

A saved group of Digital Humans that you can add to any simulation, so you reuse the same test suite instead of building it again.

Composite Metric

A custom metric that combines the results of other metrics into one number or label. It does not call an LLM.

Conditional Metric

A custom metric that runs only on conversations that meet a condition, for example when the agent called a given tool. On other conversations it has no result.

Connection Type

How Bluejay reaches your agent to call or message it. You choose it when you add the agent, and each connection type has its own setup page under Integrations.

Conversation

One call, text thread, or email thread between an agent and a caller. It includes the transcript and the metric results.

Custom Metric

A rule that you write to grade each conversation. It can grade simulations, production calls, or both.

Customer Journeys

A simulation type in which the same Digital Human contacts your agent several times in a fixed order, to test how the agent treats a customer who comes back.

Dashboard

A page of widgets that chart your metrics over time, for either simulation data or production data.

Diarization

Splitting an audio recording by speaker, so that each thing said is credited to the right person.

Digital Human

A simulated customer that calls, texts, or emails an agent in a simulation. Each one is a test case, and it runs once as each of its personas, or once as the baseline, its own settings, when it has none. A customer journey always runs as the baseline.

DTMF

The tones a phone makes when you press its keypad. A Digital Human can send them to an agent, for example to choose a menu option.

Dynamic Variable

A placeholder in a metric prompt. Before the judge runs, Bluejay replaces it with the value of that key in the conversation metadata.

Error Code

The code on a failed call that says what went wrong, for example NO_ANSWER. Each code belongs to one failed call status, such as No Connection or System Error.

Evaluate

The API you use to send one production conversation to Bluejay, which then scores it with your custom metrics.

Folder

A group of agents. A webhook can select a folder to cover all of its agents at once.

Hallucination

A built-in metric that checks what an agent says against the facts in its Knowledge Bases. Add it from the Library tab on the Metrics page.

Inbound Agent

An agent that answers conversations the customer starts. In a simulation, the Digital Human calls, texts, or emails the agent first.

Issue

A group of similar failures, or a known system error, so that you can triage them as one problem instead of one at a time. Simulations and production have separate issue lists.

IVR Flow

A phone menu that you draw on the Scenario Builder page. In the IVR Flow simulation type, the Digital Human plays the menu and your outbound agent calls through it.

Knowledge Base

A set of documents with the facts your agent must know. Bluejay uses it to grade the agent’s answers and to generate Digital Humans.

Label

A name, for example production, that points to one agent version, so a run can pick a version by name.

Latency

How long the agent takes to respond: the time from when the caller stops speaking to when the agent starts.

LLM as a Judge

A type of custom metric in which an LLM reads the conversation and scores it using a prompt that you write.

Load Testing

A simulation type that runs many conversations at the same time, to test your agent’s latency and throughput under load.

MCP Server

The Bluejay server that lets an AI assistant work in Bluejay for you. You ask in plain language, and the assistant creates simulations, runs them, and reads the results.

Metrics Lab

Where you label conversations with the verdict you expect, so that Bluejay can tune a custom metric until it agrees with you.

Mono

Audio with a single channel, as opposed to stereo. The CHIRP protocol uses mono audio.

Observability

The section of the Bluejay app for your real production conversations, where custom metrics score them and you review the results in Logs. The docs cover it under Monitor.

Outbound Agent

An agent that starts the conversation, for example by calling the customer. To test one, Bluejay usually calls your outbound trigger webhook, and your system then contacts the Digital Human.

Outbound Trigger Webhook

A URL on your system that Bluejay sends a POST to for each conversation in a run. Your system then calls or messages the Digital Human, so your outbound agent starts the conversation.

pcm_s16le

The audio encoding that the CHIRP protocol uses: signed 16-bit little-endian PCM.

Persona

A reusable caller: who calls your agent, with their personality, voice and behavior. A Digital Human runs once as each persona it lists, so one test covers several callers. Customer journeys are the exception and run as the baseline.

Personality

A field on a Digital Human: one line that says who the caller is. It applies when the Digital Human runs as the baseline, without a persona.

PSTN

The public phone network, which carries calls to regular phone numbers. It usually drops custom SIP headers.

Red Teaming

Tests in which adversarial Digital Humans call an agent to find out whether a caller can trick it into leaking data or saying harmful things.

Redundancy

A built-in metric that flags when the agent repeats itself. Add it from the Library tab on the Metrics page.

Reevaluate

Scoring an existing conversation again. No new call is made.

Regression Testing

Running the same simulation again after you change your agent, to catch anything that got worse before you ship.

Response Type

The kind of result a custom metric returns, for example Pass/Fail, a number, or one label from a list.

Sample Rate

The number of audio samples taken each second. The CHIRP protocol uses 16000 Hz.

Scenario

A conversation flow that you draw as a graph on the Scenario Builder page. Bluejay makes one Digital Human for each path through the graph.

Schedule

A setting that runs a simulation automatically at fixed times.

Scripted Response

A fixed reply that a Digital Human gives when the agent says a given line, so the reply is the same on every run.

Self-Improve

A run in which Bluejay finds why calls fail, rewrites the agent prompt, and keeps each change only if the simulation pass rate goes up.

Simulation

A set of Digital Humans that test one agent. You can run the same simulation as many times as you need.

Simulation Result

The outcome of one Digital Human’s conversation in a simulation run, including its transcript and scores.

Simulation Run

One run of a simulation, in which each Digital Human has at least one conversation with the agent.

Simulation Type

The way a simulation makes its Digital Humans, for example from a description, a CSV file, or past conversations. Each type has its own page under Simulations.

SIP

Session Initiation Protocol, a way to connect voice systems to each other directly. SIP calls can carry headers such as X-Simulation-Result-Id.

Success Criteria

What decides whether one Digital Human’s call passes or fails: a list of things the agent should do that the transcript can show. Bluejay grades each bullet, and the call passes only when all of them pass.

Telephony

Simulations that run over phone numbers. A Digital Human calls an inbound agent, or an outbound agent calls a Digital Human.

Test

The part of Bluejay for checking your agent before it reaches production, such as simulations and Red Teaming. In the app, it is the Simulations section.

Test Case

One thing a caller tries to do, with the success criteria that grade the call. In Bluejay each test case is a Digital Human, listed on the Test Cases tab.

Tool Calls

The functions an agent calls during a conversation, such as looking up an account. Send them to Bluejay so that custom metrics can check what the agent did, not only what it said.

Trace

An OpenTelemetry record of the steps your agent took behind the scenes. When you link a trace to a conversation, Bluejay shows them side by side.

Traits

Facts that a Digital Human knows and gives when the agent asks, for example a member ID. You set them on each Digital Human, and each trait has the same value on every run.

Transcript

The written text of a conversation, with each line marked by who said it.

Uptime Monitor

A test conversation that Bluejay runs with your live agent on a schedule. It notifies you when the agent goes DOWN and again when it is back UP.

Webhook

A URL on your server that Bluejay sends a signed POST request to when an event occurs, for example when a simulation ends.

Workflow

The multi-agent graph of an agent, for example a root agent that hands off to sub-agents. You can pull it from your provider, edit it, and push it back, or import one from a document. It is different from a scenario.

Last modified on November 16, 2026