> ## Documentation Index
> Fetch the complete documentation index at: https://docs.getbluejay.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# GitHub

> Fail a build when your agent's score drops.

Run a simulation after each deploy or pull request through the Bluejay GitHub App or the GitHub Action.

## Set up the GitHub App

**1. Connect GitHub**

Go to **Settings > Integrations**, open the **CI/CD** tab, select **GitHub**, click **Connect**, and approve the Bluejay GitHub App on the repositories you want to test.

**2. Add a trigger**

Open a simulation, click **CI/CD**, and add a trigger with a repository, an environment name, and a minimum score. The environment name must match the deployment environment your host reports to GitHub, for example `Production`.

The host must report deployments to GitHub. Vercel, Netlify, Railway and Render do. If yours does not, [use the GitHub Action](#use-the-github-action-instead).

Each run uses credits.

## Production and preview triggers

A **Production** trigger runs the simulation against the saved agent after a successful production deploy.

A preview trigger runs against the preview deployment, for that one run only. The saved agent is not changed. Set an endpoint template, and Bluejay fills in the placeholders from the deployment:

| Placeholder | Value |
| - | - |
| | Full URL of the deployment |
| | Host of the deployment URL |
| | Commit SHA |
| | First 7 characters of the SHA |
| | Pull request number. The run fails to queue if the commit has no pull request |
| | Environment name |

Examples:

* WebSocket agent: `wss://{environment_url_host}/ws`
* LiveKit agent: `my-agent-pr-{pr_number}`, used as the LiveKit agent name

Preview runs work for agents that connect by WebSocket, HTTP webhook, OpenAI chat, OpenAI Realtime, LiveKit or Pipecat. Phone and SIP agents, and agents hosted on builders such as Vapi, Retell and ElevenLabs, only support Production triggers.

Set a **Preview domain** such as `vercel.app` when the template uses  or . It is required for those templates. The host of the rendered URL must end with it. Only `https` and `wss` URLs are allowed, and private addresses are blocked.

If previews are protected, for example by Vercel Deployment Protection, add a bypass header to the trigger so Bluejay can reach the deployment.

## See the result

Bluejay adds a check to the commit and its pull request. It passes when the score is at least the trigger's minimum score, and links to the run. To block merges on a failing score, make the check required in your branch protection rules.

## Use the GitHub Action instead

Use the Action when your host does not report deployments to GitHub, or when you want the run inside your own workflow. You need a Bluejay [API key](https://docs.getbluejay.ai/api-reference/introduction.md#make-your-first-request), an agent, and a simulation with the Digital Humans you want to run.

### Test an existing agent

All runs go to the same simulation, so you can compare them.

**1. Add your API key and variables**

In GitHub, under **Settings > Secrets and variables > Actions**, add your API key as a repository secret named `BLUEJAY_API_KEY`, and add these repository variables:

| Variable name | Value | Required |
| - | - | - |
| BLUEJAY\_SIMULATION\_ID | Your simulation ID (e.g., `12345`) | Yes |
| BLUEJAY\_MIN\_SCORE | Minimum score (e.g., `80`) | No |
| BLUEJAY\_PROMPT\_ID | Agent version override ID (the action input is `prompt_id`) | No |
| BLUEJAY\_KB\_ID | Knowledge Base override ID | No |
| BLUEJAY\_DIGITAL\_HUMAN\_IDS | Comma-separated Digital Human IDs | No |

The simulation ID is in the simulation's **General Settings** tab, and each Digital Human ID is on its card in the **Digital Humans** tab.

**2. Create your workflow**

Add `.github/workflows/bluejay-tests.yml` to your repo:

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
name: Agent Tests

on:
  workflow_dispatch:
    inputs:
      simulation_id:
        description: 'Bluejay Simulation ID to run'
        required: false
        type: string
      min_score:
        description: 'Minimum score (0-100)'
        required: false
        type: string
        default: '80'
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize]

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          # Required: simulation id (manual input OR repo variable)
          simulation_id: ${{ inputs.simulation_id || vars.BLUEJAY_SIMULATION_ID }}
          # Optional overrides (omit the ones you do not use)
          prompt_id: ${{ vars.BLUEJAY_PROMPT_ID }}
          knowledge_base_id: ${{ vars.BLUEJAY_KB_ID }}
          digital_human_ids: ${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}
          # Behavior controls
          wait_for_results: 'true'
          min_score: ${{ inputs.min_score || vars.BLUEJAY_MIN_SCORE || '80' }}
          poll_interval_seconds: '10'
          timeout_seconds: '1500'
```

**3. Trigger a simulation**

Open a pull request. The action runs the simulation on each PR and shows pass or fail in the PR checks. The score is in the job logs.

![GitHub Action Check](https://mintcdn.com/bluejay/13QTZFHWYyYMSQI8/images/trigger_github_action.png?fit=max\&auto=format\&n=13QTZFHWYyYMSQI8\&q=85\&s=7fe8d0c5156098b39f9dfb33ed280524)

### Test a preview deployment

Use this option if CI makes a preview deployment for each PR. For each PR, the workflow creates an agent and a simulation, runs the test, and then deletes both. Your Digital Humans are not deleted, because they are your test suite and every PR reuses them.

The example uses a `WEBSOCKET` agent. Set `PREVIEW_WEBSOCKET_URL` to the URL of the preview deployment. For other connection types, send that type's fields instead. [Add Agent](https://docs.getbluejay.ai/api-reference/endpoint/add-agent.md) lists them.

For this option, set `BLUEJAY_DIGITAL_HUMAN_IDS`, for example `101,102`, because a new simulation has no Digital Humans and the run fails without them.

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
name: Agent Tests (ephemeral)

on:
  pull_request:
    types: [opened, synchronize]

env:
  BLUEJAY_API: https://api.getbluejay.ai/v1
  BLUEJAY_API_KEY: ${{ secrets.BLUEJAY_API_KEY }}
  PREVIEW_WEBSOCKET_URL: wss://pr-${{ github.event.number }}.preview.example.com/ws

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Create ephemeral agent
        id: agent
        run: |
          payload=$(jq -n \
            --arg name "ci-pr-${{ github.event.number }}-run-${{ github.run_id }}" \
            --arg ws_url "$PREVIEW_WEBSOCKET_URL" \
            '{
              name: $name,
              system_prompt: "Customer support assistant under test",
              goals: ["Resolve the customer issue"],
              connection_type: "WEBSOCKET",
              websocket_url: $ws_url
            }')
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/add-agent" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d "$payload")
          echo "id=$(echo "$resp" | jq -r '.agent_id')" >> "$GITHUB_OUTPUT"

      - name: Create simulation
        id: sim
        run: |
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/create-simulation" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "agent_id": "${{ steps.agent.outputs.id }}",
              "name": "CI PR #${{ github.event.number }}"
            }')
          echo "id=$(echo "$resp" | jq -r '.simulation_id')" >> "$GITHUB_OUTPUT"

      - name: Attach Digital Humans
        run: |
          curl -s --fail-with-body -X POST "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}/add-digital-humans" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"digital_human_ids": [${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}]}'

      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          simulation_id: ${{ steps.sim.outputs.id }}
          wait_for_results: 'true'
          min_score: '80'

      - name: Clean up
        if: always()
        run: |
          status=0
          if [ -n "${{ steps.sim.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          if [ -n "${{ steps.agent.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/agent/${{ steps.agent.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          exit $status
```

> [!WARNING]
>
> The cleanup step deletes the agent and the simulation, and the run history with them. To keep results, remove the cleanup step or [test an existing agent](#test-an-existing-agent).

### Pin to a commit SHA

Some organizations require every third-party action to be pinned to a full-length commit SHA. If yours does, the workflow fails with *"all actions must be pinned to a full-length commit SHA"*. To find the commit for `v1`, run:

```bash theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
git ls-remote https://github.com/bluejay-ai-dev/bluejay-github-actions 'refs/tags/v1^{}'
```

Then put the SHA in place of `v1`:

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
uses: bluejay-ai-dev/bluejay-github-actions@c28260e8bd21f29e3841c712131a0ae979c7bdd7 # v1
```

> [!NOTE]
>
> A SHA pin does not follow updates to `v1`. To get fixes, run the command again and update the pin.

### Inputs

| Input | Required | Default | Description |
| - | - | - | - |
| `api_key` | Yes | None | Your Bluejay API key. |
| simulation\_id | Yes | None | ID of the simulation to run. |
| `prompt_id` | No | None | Agent version UUID to use for this run. |
| knowledge\_base\_id | No | None | Override Knowledge Base for this run. |
| digital\_human\_ids | No | None | Comma-separated list of Digital Human IDs. |
| phone\_number | No | None | Phone number to use for this run. |
| `sip_uri` | No | None | SIP URI to use for this run. |
| wait\_for\_results | No | `true` | If `true`, wait for the run to end and fail CI on the score. |
| `min_score` | No | `80` | Minimum score (0-100). |
| poll\_interval\_seconds | No | `10` | Polling frequency in seconds. |
| timeout\_seconds | No | `1500` | Maximum wait in seconds (25 minutes). |

The step fails if the run does not complete, if the score is below `min_score`, or if `timeout_seconds` passes first. To only queue the run without checking the score, set `wait_for_results` to `false`.

### Outputs

| Output | Description |
| - | - |
| `simulation-run-id` | The ID of the queued simulation run. |
| `final-status` | The final run status from the API, for example `completed`. |
| `score` | Score of the run: `tests_passed / total_tests × 100`. |

### Triggers

To run the tests on a different event, change only the `on:` section. Any [GitHub Actions trigger](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows) works. For example, to run daily at 09:00 UTC:

```yaml theme={"theme":{"light":"vitesse-light","dark":"vitesse-dark"}}
on:
  schedule:
    - cron: '0 9 * * *'
```

## Related

* [Queue Simulation Run](https://docs.getbluejay.ai/api-reference/endpoint/queue-simulation-run.md)
* [Simulation](https://docs.getbluejay.ai/test/simulations/create.md)
* [Simulation Results](https://docs.getbluejay.ai/test/simulations/create.md#read-the-results)
* [Bluejay as Code](https://docs.getbluejay.ai/bluejay-as-code/overview.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.