GitHub Actions

Run Bluejay simulations in GitHub Actions and fail CI if your agent doesn’t meet quality standards. Just like unit tests, but for AI agents. Key capabilities:

Fail CI if your pass rate drops below your threshold
Automatically test every PR and commit
Zero-install; runs on GitHub-hosted runners
Override prompts, knowledge bases, and Digital Humans per run

How It Works

The action is a thin client over the Bluejay REST API:

It queues a run for the simulation you specify (POST /v1/queue-simulation-run).
It polls until the run finishes (GET /v1/retrieve-simulation-results/{run_id}).
It computes the score as the pass rate: tests_passed / total_tests × 100.
The step fails if the run doesn’t complete successfully, or the score is below min_score.

Before Starting

You’ll need:

Bluejay API Key – Get yours from the Bluejay dashboard
An agent – Connected to your assistant
A simulation – Attached to that agent, with the Digital Humans (test scenarios) you want to run

The agent you configure in Bluejay owns the connection to your assistant, so the CI workflow is the same regardless of how your agent is connected: pass an api_key and a simulation_id. There are two integration patterns. Most teams should start with Option 1.

Option 1: Test an Existing Agent (Recommended)

Configure the agent and simulation once — in the dashboard or via the API — and CI only references the simulation. All run history accumulates on one simulation, so you can compare runs over time in the dashboard.

Add your API key and variables

Go to: Settings → Secrets and variables → ActionsAdd a Secret:

Click New repository secret
Name: BLUEJAY_API_KEY
Value: Your API key from the developers page

Add Variables:

Click the Variables tab
Click New repository variable
Add the following:

Variable Name	Value	Required
`BLUEJAY_SIMULATION_ID`	Your simulation ID (e.g., `12345`)	Yes
`BLUEJAY_MIN_SCORE`	Minimum passing score (e.g., `80`)	No
`BLUEJAY_PROMPT_ID`	Agent version override ID (the action input is still named `prompt_id`)	No
`BLUEJAY_KB_ID`	Knowledge base override ID	No
`BLUEJAY_DIGITAL_HUMAN_IDS`	Comma-separated Digital Human IDs	No

Create your workflow

Add .github/workflows/bluejay-tests.yml to your repo:

name: Agent Tests

on:
  workflow_dispatch:
    inputs:
      simulation_id:
        description: 'Bluejay Simulation ID to run'
        required: false
        type: string
      min_score:
        description: 'Minimum required score (0–100)'
        required: false
        type: string
        default: '80'
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize]

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          # Required: simulation id (manual input OR repo variable)
          simulation_id: ${{ inputs.simulation_id || vars.BLUEJAY_SIMULATION_ID }}
          # Optional overrides (omit any you don't use)
          prompt_id: ${{ vars.BLUEJAY_PROMPT_ID }}
          knowledge_base_id: ${{ vars.BLUEJAY_KB_ID }}
          digital_human_ids: ${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}
          # Behavior controls
          wait_for_results: 'true'
          min_score: ${{ inputs.min_score || vars.BLUEJAY_MIN_SCORE || '80' }}
          poll_interval_seconds: '10'
          timeout_seconds: '1500'

Trigger a simulation

Make changes to your codebase and open a pull request. The GitHub Action will automatically run Bluejay tests on every PR.

Monitor your simulation

Click on the Actions tab in your GitHub repository to view the simulation run in real-time. You’ll see the status and score once the simulation completes.

Option 2: Fully Stateless (Create → Test → Delete)

If your CI spins up an ephemeral deployment per PR (a preview environment with its own URL), you can create the Bluejay agent and simulation inside the workflow, run the tests, and delete everything afterwards. Your Digital Humans stay persistent — they are your test suite — while the agent and simulation are created fresh each run. In the add-agent payload, include the connection field that matches how Bluejay reaches your preview deployment — see the add-agent API reference for the available fields. The agent’s connection type is inferred from whichever field you provide. Set the BLUEJAY_DIGITAL_HUMAN_IDS repository variable (e.g. 101,102) before using this pattern — unlike Option 1, it is required here: a freshly created simulation has no Digital Humans attached, and the run fails without them.

name: Agent Tests (ephemeral)

on:
  pull_request:
    types: [opened, synchronize]

env:
  BLUEJAY_API: https://api.getbluejay.ai/v1
  BLUEJAY_API_KEY: ${{ secrets.BLUEJAY_API_KEY }}

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Create ephemeral agent
        id: agent
        run: |
          payload=$(jq -n \
            --arg name "ci-pr-${{ github.event.number }}-run-${{ github.run_id }}" \
            --arg connection "<how Bluejay reaches this PR's preview deployment>" \
            '{
              name: $name,
              system_prompt: "Customer support assistant under test",
              knowledge_base: "",
              goals: ["Resolve the customer issue"],
              "<your connection field>": $connection
            }')
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/add-agent" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d "$payload")
          echo "id=$(echo "$resp" | jq -r '.agent_id')" >> "$GITHUB_OUTPUT"

      - name: Create simulation
        id: sim
        run: |
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/create-simulation" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "agent_id": "${{ steps.agent.outputs.id }}",
              "name": "CI PR #${{ github.event.number }}"
            }')
          echo "id=$(echo "$resp" | jq -r '.simulation_id')" >> "$GITHUB_OUTPUT"

      - name: Attach Digital Humans
        run: |
          curl -s --fail-with-body -X POST "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}/add-digital-humans" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"digital_human_ids": [${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}]}'

      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          simulation_id: ${{ steps.sim.outputs.id }}
          wait_for_results: 'true'
          min_score: '80'

      - name: Clean up
        if: always()
        run: |
          status=0
          if [ -n "${{ steps.sim.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          if [ -n "${{ steps.agent.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/agent/${{ steps.agent.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          exit $status

Deleting the agent and simulation removes the run’s history from your dashboard. If you want results you can browse and compare later, skip the cleanup step — or use Option 1 with a persistent agent, which is what most teams want.

Pinning to a Commit SHA

Some organizations require third-party actions to be pinned to a full-length commit SHA instead of a tag (you’ll see an error like “all actions must be pinned to a full-length commit SHA”). Tags such as v1 are movable pointers; a commit SHA is immutable, so pinning it guarantees the exact code your pipeline runs. The action’s repository is public, so you can resolve the commit v1 points to yourself:

git ls-remote https://github.com/bluejay-ai-dev/bluejay-github-actions 'refs/tags/v1^{}'

(Or on GitHub: open the repository, click Tags, and copy the commit for v1.) Then use the SHA in your workflow:

uses: bluejay-ai-dev/bluejay-github-actions@c28260e8bd21f29e3841c712131a0ae979c7bdd7 # v1

The SHA above is what v1 resolves to as of July 2026.

SHA pins don’t follow tag updates: when we ship fixes to v1, re-run the command above and bump your pin. Dependabot with package-ecosystem: github-actions can do this for you automatically.

Inputs

Input	Required	Default	Description
`api_key`	Yes	—	Your Bluejay API key.
`simulation_id`	Yes	—	ID of the simulation to run.
`prompt_id`	No	—	Deprecated. Same uuid as `agent_version_id`. Override the agent version for this run.
`knowledge_base_id`	No	—	Override knowledge base for this run.
`digital_human_ids`	No	—	Comma-separated list of Digital Human IDs.
`wait_for_results`	No	`true`	Wait for simulation to finish.
`min_score`	No	`80`	Required pass rate (0–100).
`poll_interval_seconds`	No	`10`	Polling frequency in seconds.
`timeout_seconds`	No	`1500`	Timeout (25 minutes).

Outputs

Output	Description
`simulation-run-id`	The ID of the queued simulation run.
`final-status`	Final simulation status: `completed`, `failed`, `cancelled`, etc.
`score`	Pass rate of the run: `tests_passed / total_tests × 100`.

Customize When Tests Run

Any GitHub Actions trigger works — only the on: section changes; the job itself stays the same.

Push Only
Pull Requests Only
Scheduled Runs
Manual

on:
  push:
    branches: [main, production, staging]

on:
  pull_request:
    types: [opened, synchronize]

Daily at 9 AM UTC:

on:
  schedule:
    - cron: '0 9 * * *'

on:
  workflow_dispatch:
    inputs:
      simulation_id:
        description: 'Bluejay Simulation ID to run'
        required: false
        type: string

Getting Started

Key Concepts

Test

Monitor

Integrations

How It Works

Before Starting

Option 1: Test an Existing Agent (Recommended)

Option 2: Fully Stateless (Create → Test → Delete)

Pinning to a Commit SHA

Inputs

Outputs

Customize When Tests Run

​How It Works

​Before Starting

​Option 1: Test an Existing Agent (Recommended)

​Option 2: Fully Stateless (Create → Test → Delete)

​Pinning to a Commit SHA

​Inputs

​Outputs

​Customize When Tests Run

How It Works

Before Starting

Option 1: Test an Existing Agent (Recommended)

Option 2: Fully Stateless (Create → Test → Delete)

Pinning to a Commit SHA

Inputs

Outputs

Customize When Tests Run