CI/CD Integration

Add the Bluejay action to a GitHub workflow to run a simulation from CI.

Prerequisites

A Bluejay API key.
An agent connected to your assistant.
A simulation on that agent, with the Digital Humans you want to run.

However your agent connects to Bluejay, you only pass api_key and simulation_id to the CI workflow.

Test an existing agent

Use this option unless CI makes a preview deployment for each PR. All runs go to the same simulation, so you can compare them.

Add your API key and variables

In GitHub, under Settings > Secrets and variables > Actions, add your API key as a repository secret named BLUEJAY_API_KEY, and add these repository variables:

Variable name	Value	Required
	Your simulation ID (e.g., `12345`)	Yes
	Minimum passing score (e.g., `80`)	No
	Agent version override ID (the action input is `prompt_id`)	No
	Knowledge Base override ID	No
	Comma-separated Digital Human IDs	No

The simulation ID is in the simulation’s General Settings tab, and each Digital Human ID is on its card in the Digital Humans tab.

Create your workflow

Add .github/workflows/bluejay-tests.yml to your repo:

name: Agent Tests

on:
  workflow_dispatch:
    inputs:
      simulation_id:
        description: 'Bluejay Simulation ID to run'
        required: false
        type: string
      min_score:
        description: 'Minimum required score (0–100)'
        required: false
        type: string
        default: '80'
  push:
    branches: [main]
  pull_request:
    types: [opened, synchronize]

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          # Required: simulation id (manual input OR repo variable)
          simulation_id: ${{ inputs.simulation_id || vars.BLUEJAY_SIMULATION_ID }}
          # Optional overrides (omit the ones you do not use)
          prompt_id: ${{ vars.BLUEJAY_PROMPT_ID }}
          knowledge_base_id: ${{ vars.BLUEJAY_KB_ID }}
          digital_human_ids: ${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}
          # Behavior controls
          wait_for_results: 'true'
          min_score: ${{ inputs.min_score || vars.BLUEJAY_MIN_SCORE || '80' }}
          poll_interval_seconds: '10'
          timeout_seconds: '1500'

Trigger a simulation

Open a pull request. The action runs the simulation on each PR and shows the status and score in the PR checks.

Test a preview deployment

Use this option if CI makes a preview deployment for each PR. For each PR, the workflow creates an agent and a simulation, runs the test, and then deletes both. Your Digital Humans are not deleted, because they are your test suite and every PR reuses them. In the add-agent payload, replace <your connection field> with the field that tells Bluejay how to reach your preview deployment. Add Agent lists the connection fields. For this option, set BLUEJAY_DIGITAL_HUMAN_IDS, for example 101,102, because a new simulation has no Digital Humans and the run fails without them.

name: Agent Tests (ephemeral)

on:
  pull_request:
    types: [opened, synchronize]

env:
  BLUEJAY_API: https://api.getbluejay.ai/v1
  BLUEJAY_API_KEY: ${{ secrets.BLUEJAY_API_KEY }}

jobs:
  run-bluejay-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Create ephemeral agent
        id: agent
        run: |
          payload=$(jq -n \
            --arg name "ci-pr-${{ github.event.number }}-run-${{ github.run_id }}" \
            --arg connection "<how Bluejay reaches this PR's preview deployment>" \
            '{
              name: $name,
              system_prompt: "Customer support assistant under test",
              goals: ["Resolve the customer issue"],
              "<your connection field>": $connection
            }')
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/add-agent" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d "$payload")
          echo "id=$(echo "$resp" | jq -r '.agent_id')" >> "$GITHUB_OUTPUT"

      - name: Create simulation
        id: sim
        run: |
          resp=$(curl -s --fail-with-body -X POST "$BLUEJAY_API/create-simulation" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{
              "agent_id": "${{ steps.agent.outputs.id }}",
              "name": "CI PR #${{ github.event.number }}"
            }')
          echo "id=$(echo "$resp" | jq -r '.simulation_id')" >> "$GITHUB_OUTPUT"

      - name: Attach Digital Humans
        run: |
          curl -s --fail-with-body -X POST "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}/add-digital-humans" \
            -H "X-API-Key: $BLUEJAY_API_KEY" \
            -H "Content-Type: application/json" \
            -d '{"digital_human_ids": [${{ vars.BLUEJAY_DIGITAL_HUMAN_IDS }}]}'

      - name: Run Bluejay Tests
        uses: bluejay-ai-dev/bluejay-github-actions@v1
        with:
          api_key: ${{ secrets.BLUEJAY_API_KEY }}
          simulation_id: ${{ steps.sim.outputs.id }}
          wait_for_results: 'true'
          min_score: '80'

      - name: Clean up
        if: always()
        run: |
          status=0
          if [ -n "${{ steps.sim.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/simulation/${{ steps.sim.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          if [ -n "${{ steps.agent.outputs.id }}" ]; then
            curl -s --fail-with-body -X DELETE "$BLUEJAY_API/agent/${{ steps.agent.outputs.id }}" \
              -H "X-API-Key: $BLUEJAY_API_KEY" || status=1
          fi
          exit $status

The cleanup step deletes the agent and the simulation, and the run history with them. To keep results, remove the cleanup step or test an existing agent.

Pin to a commit SHA

Some organizations require every third-party action to be pinned to a full-length commit SHA. If yours does, the workflow fails with “all actions must be pinned to a full-length commit SHA”. To find the commit for v1, run:

git ls-remote https://github.com/bluejay-ai-dev/bluejay-github-actions 'refs/tags/v1^{}'

Then put the SHA in place of v1:

uses: bluejay-ai-dev/bluejay-github-actions@c28260e8bd21f29e3841c712131a0ae979c7bdd7 # v1

A SHA pin does not follow updates to v1. To get fixes, run the command again and update the pin.

Inputs

Input	Required	Default	Description
`api_key`	Yes	None	Your Bluejay API key.
	Yes	None	ID of the simulation to run.
`prompt_id`	No	None	Agent version UUID to use for this run.
	No	None	Override Knowledge Base for this run.
	No	None	Comma-separated list of Digital Human IDs.
	No	None	Phone number to use for this run.
`sip_uri`	No	None	SIP URI to use for this run.
	No	`true`	If `true`, wait for the run to end and fail CI on the score.
`min_score`	No	`80`	Required pass rate (0–100).
	No	`10`	Polling frequency in seconds.
	No	`1500`	Maximum wait in seconds (25 minutes).

The step fails if the run does not complete, if the pass rate is below min_score, or if timeout_seconds passes first. To only queue the run without checking the score, set wait_for_results to false.

Outputs

Output	Description
`simulation-run-id`	The ID of the queued simulation run.
`final-status`	The final run status from the API, for example `completed`.
`score`	Pass rate of the run: `tests_passed / total_tests × 100`.

Triggers

To run the tests on a different event, change only the on: section. Any GitHub Actions trigger works. For example, to run daily at 09:00 UTC:

on:
  schedule:
    - cron: '0 9 * * *'

Overview

Test

Monitor

Improve

Configure

Prerequisites

Test an existing agent

Test a preview deployment

Pin to a commit SHA

Inputs

Outputs

Triggers

​Prerequisites

​Test an existing agent

​Test a preview deployment

​Pin to a commit SHA

​Inputs

​Outputs

​Triggers

​Related

Prerequisites

Test an existing agent

Test a preview deployment

Pin to a commit SHA

Inputs

Outputs

Triggers

Related