Observability Alerts

Alerts notify your team when production metrics cross a boundary repeatedly within a time window. Instead of manually checking dashboards, alerts push signals to Slack or email the moment a sustained problem appears.

​

How Alerts Work

After every production call is scored through the observability pipeline, Bluejay checks whether any metric has crossed its configured threshold. Violations are counted over a rolling time window, and the alert only fires when enough violations accumulate — filtering out transient noise while catching real problems.

​

Configuration

​

Example: Latency Spike Detection

An alert configured to fire when average agent latency exceeds 3 seconds at least 5 times within 10 minutes:

• A single slow call does not trigger the alert — one violation is below the required 5.
• Three slow calls in 10 minutes still does not trigger — only 3 of 5 occurrences reached.
• A fifth slow call within the same 10-minute window fires the alert and notifies your team.
• If 20 minutes pass with only 2 violations, earlier ones roll out of the window — no alert.

For zero-tolerance metrics like hallucination detection, set occurrences to 1 so the alert fires on the first violation.

​

Routing to Slack

Connect Bluejay to your Slack workspace through the Slack integration and route alerts to specific channels:

• Engineering — latency spikes, error rate increases, model regressions
• Support/QA — hallucination detections, compliance failures, low satisfaction
• Leadership — quality summaries, sustained threshold breaches

To kick off an automated flow when the alert fires (triage, ticket, workflow), subscribe a URL under Settings → Webhooks to Observability alert triggered. That is separate from Slack/email on the alert itself.

​

Common Configurations

​

Best Practices

• Start with a few high-signal alerts — avoid alert fatigue by focusing on metrics that drive the most business impact
• Route alerts to the team that can act — engineering alerts go to engineering channels, quality alerts go to QA
• Review and tune thresholds regularly — as your agent improves, tighten thresholds to maintain a rising quality bar
• Pair alerts with dashboards — alerts tell you something happened; dashboards help you investigate why

​

Next Steps