The voice, text, or email AI system that you want Bluejay to test and monitor. Your simulations, conversations, and custom metrics all belong to an agent.
Values that every Digital Human of an agent knows, for example a business name. A Digital Human’s trait with the same name overrides one.
A saved, numbered copy of an agent prompt. Labels such as production and latest point to a version and decide which one a run uses.
A rule that notifies your team when a metric on production conversations crosses a threshold.
A WebSocket message that carries raw bytes instead of text. In the CHIRP protocol, binary frames carry the audio.
The assistant inside the Bluejay app. It tells you why calls failed and can make the change for you, for example create a simulation, a custom metric, or a dashboard.
Keeping your agents, simulations, Digital Humans, and custom metrics in a YAML file. You pull the file from Bluejay, edit it, and push it back, by hand or from CI.
Where a simulated call is in its life, for example Queued, Running, or Completed. A call that fails also has an error code that says why.
The protocol that Bluejay uses to exchange audio and text events with your server over a WebSocket.
A saved group of Digital Humans that you can add to any simulation, so you reuse the same test suite instead of building it again.
A custom metric that combines the results of other metrics into one number or label. It does not call an LLM.
A custom metric that runs only on conversations that meet a condition, for example when the agent called a given tool. On other conversations it has no result.
How Bluejay reaches your agent to call or message it. You choose it when you add the agent, and each connection type has its own setup page under Integrations.
One call, text thread, or email thread between an agent and a caller. It includes the transcript and the metric results.
A rule that you write to grade each conversation. It can grade simulations, production calls, or both.
A simulation type in which the same Digital Human contacts your agent several times in a fixed order, to test how the agent treats a customer who comes back.
A page of widgets that chart your metrics over time, for either simulation data or production data.
Splitting an audio recording by speaker, so that each thing said is credited to the right person.
A simulated customer that calls, texts, or emails an agent in a simulation. Each one is a test case, and it runs once as each of its personas, or once as the baseline, its own settings, when it has none. A customer journey always runs as the baseline.
The tones a phone makes when you press its keypad. A Digital Human can send them to an agent, for example to choose a menu option.
A placeholder in a metric prompt. Before the judge runs, Bluejay replaces it with the value of that key in the conversation metadata.
The code on a failed call that says what went wrong, for example NO_ANSWER. Each code belongs to one failed call status, such as No Connection or System Error.
The API you use to send one production conversation to Bluejay, which then scores it with your custom metrics.
A group of agents. A webhook can select a folder to cover all of its agents at once.
A built-in metric that checks what an agent says against the facts in its Knowledge Bases. Add it from the Library tab on the Metrics page.
An agent that answers conversations the customer starts. In a simulation, the Digital Human calls, texts, or emails the agent first.
A group of similar failures, or a known system error, so that you can triage them as one problem instead of one at a time. Simulations and production have separate issue lists.
A set of documents with the facts your agent must know. Bluejay uses it to grade the agent’s answers and to generate Digital Humans.
A name, for example production, that points to one agent version, so a run can pick a version by name.
How long the agent takes to respond: the time from when the caller stops speaking to when the agent starts.
A type of custom metric in which an LLM reads the conversation and scores it using a prompt that you write.
A simulation type that runs many conversations at the same time, to test your agent’s latency and throughput under load.
The Bluejay server that lets an AI assistant work in Bluejay for you. You ask in plain language, and the assistant creates simulations, runs them, and reads the results.
Where you label conversations with the verdict you expect, so that Bluejay can tune a custom metric until it agrees with you.
Audio with a single channel, as opposed to stereo. The CHIRP protocol uses mono audio.
The section of the Bluejay app for your real production conversations, where custom metrics score them and you review the results in Logs. The docs cover it under Monitor.
An agent that starts the conversation, for example by calling the customer. To test one, Bluejay usually calls your outbound trigger webhook, and your system then contacts the Digital Human.
A URL on your system that Bluejay sends a POST to for each conversation in a run. Your system then calls or messages the Digital Human, so your outbound agent starts the conversation.
The audio encoding that the CHIRP protocol uses: signed 16-bit little-endian PCM.
A reusable caller: who calls your agent, with their personality, voice and behavior. A Digital Human runs once as each persona it lists, so one test covers several callers. Customer journeys are the exception and run as the baseline.
A field on a Digital Human: one line that says who the caller is. It applies when the Digital Human runs as the baseline, without a persona.
The public phone network, which carries calls to regular phone numbers. It usually drops custom SIP headers.
Tests in which adversarial Digital Humans call an agent to find out whether a caller can trick it into leaking data or saying harmful things.
A built-in metric that flags when the agent repeats itself. Add it from the Library tab on the Metrics page.
Running the same simulation again after you change your agent, to catch anything that got worse before you ship.
The kind of result a custom metric returns, for example Pass/Fail, a number, or one label from a list.
The number of audio samples taken each second. The CHIRP protocol uses 16000 Hz.
A conversation flow that you draw as a graph on the Scenario Builder page. Bluejay makes one Digital Human for each path through the graph.
A fixed reply that a Digital Human gives when the agent says a given line, so the reply is the same on every run.
A run in which Bluejay finds why calls fail, rewrites the agent prompt, and keeps each change only if the simulation pass rate goes up.
A set of Digital Humans that test one agent. You can run the same simulation as many times as you need.
The outcome of one Digital Human’s conversation in a simulation run, including its transcript and scores.
One run of a simulation, in which each Digital Human has at least one conversation with the agent.
The way a simulation makes its Digital Humans, for example from a description, a CSV file, or past conversations. Each type has its own page under Simulations.
Session Initiation Protocol, a way to connect voice systems to each other directly. SIP calls can carry headers such as X-Simulation-Result-Id.
What decides whether one Digital Human’s call passes or fails: a list of things the agent should do that the transcript can show. Bluejay grades each bullet, and the call passes only when all of them pass.
Simulations that run over phone numbers. A Digital Human calls an inbound agent, or an outbound agent calls a Digital Human.
The part of Bluejay for checking your agent before it reaches production, such as simulations and Red Teaming. In the app, it is the Simulations section.
One thing a caller tries to do, with the success criteria that grade the call. In Bluejay each test case is a Digital Human, listed on the Test Cases tab.
The functions an agent calls during a conversation, such as looking up an account. Send them to Bluejay so that custom metrics can check what the agent did, not only what it said.
An OpenTelemetry record of the steps your agent took behind the scenes. When you link a trace to a conversation, Bluejay shows them side by side.
Facts that a Digital Human knows and gives when the agent asks, for example a member ID. You set them on each Digital Human, and each trait has the same value on every run.
A test conversation that Bluejay runs with your live agent on a schedule. It notifies you when the agent goes DOWN and again when it is back UP.
A URL on your server that Bluejay sends a signed POST request to when an event occurs, for example when a simulation ends.