What You’ll Learn
- What Alerts are and how they fit into monitoring workflows
- How to configure threshold-based notifications
- How Alerts connect to Slack and other integrations
How Alerts Work
Bluejay uses threshold alerts that count how many times a metric crosses a boundary within a rolling time window. The alert only fires when the violation count reaches the required number of occurrences — filtering out noise while catching sustained problems.Threshold Alert Configuration
When creating a threshold alert, you configure the metric to watch, the boundary condition, and the sensitivity of the trigger:
For example, an alert on “Average Agent Latency > 3 seconds” with 5 occurrences in a 10-minute window will only fire when 5 calls exceed 3-second latency within that window. A single slow call won’t trigger a notification — but a sustained regression will.
For critical metrics like hallucination detection, set occurrences to 1 so the alert fires on the very first violation.
Key Capabilities
- Threshold-based triggers — fire alerts when a metric crosses a boundary a configured number of times within a time window
- Configurable sensitivity — tune the number of occurrences and the time window to match the severity of each metric
- Channel routing — send alerts to specific Slack channels so the right team gets the right signal, or subscribe a URL under Settings → Webhooks to kick off an automated flow when an observability or simulation alert fires
- Dashboard integration — alert badges appear on agent dashboards for at-a-glance health monitoring
- Works for both observability and simulations — monitor production calls and test runs with the same alert model
Common Use Cases
- Alert engineering when average latency exceeds 3 seconds across 5 calls in 10 minutes
- Notify the support team the moment any production call is flagged for hallucination (occurrences set to 1)
- Alert QA when simulation goal completion drops below 85% across 3 conversations in a run
- Route compliance violations to a dedicated Slack channel with zero-tolerance thresholds
Next Steps
Observability Alerts
Configure threshold alerts for production monitoring.
Simulation Alerts
Configure threshold alerts for simulation testing.
Slack Integration
Connect Bluejay to Slack for real-time alert delivery.
Custom Metrics
Define the metrics that power your alert thresholds.
Issues
Recurring failures grouped for triage, not threshold notifications.