Testing Runbooks

A runbook test runs the runbook’s blocks from the command line and checks their results. You describe the test cases in a runbook_test.yml file next to runbook.mdx, then run runbooks-cli test /path/to/runbook, or runbooks-cli test ./... for every runbook under the current directory. Tests run locally or in CI.

The test runner is a standalone CLI in the Runbooks repository. From a checkout, just compile-test-cli builds it into a single binary at resources/bin/runbooks-test, which runs without Node.js or Bun. You can also run it from source with bun cli/index.ts test. The examples below use runbooks-cli as shorthand for either invocation.

Quick start

1. Create a runbook_test.yml file next to your runbook.mdx. This minimal configuration runs every block in document order and expects each one to succeed:

version: 1

tests:
  - name: happy-path

Add inputs, steps, and assertions as you need them. For a fuller example to start from, see testdata/sample-runbooks/my-first-runbook/runbook_test.yml in the Runbooks repository.

2. Run the tests:

runbooks-cli test ./my-runbook

Test configuration

A fuller runbook_test.yml:

version: 1

settings:
  # use_temp_working_dir: true  # Default: use isolated temp directory
  # working_dir: .              # Alternative: use runbook's directory
  # output_path: generated      # Where to write generated files (relative to working_dir)
  timeout: 5m            # Limit for each script and git command
  parallelizable: true   # Can run in parallel with other runbooks

tests:
  - name: happy-path
    description: Standard successful execution

    inputs:
      project.Name: "test-project"
      project.Language: "python"

    steps:
      - block: check-requirements
        expect: success

      - block: create-project
        expect: success

    assertions:
      - type: file_exists
        path: README.md

      - type: file_contains
        path: README.md
        contains: "test-project"

    cleanup:
      - command: rm -rf /tmp/test-resources

Top-level structure

version: 1              # Required. Must be 1.
settings: { ... }       # Optional. Global settings for all tests.
tests: [ ... ]          # Required. Array of test cases (minimum 1).

Field	Type	Required	Description
version	integer	Yes	Configuration version. Must be 1.
settings	object	No	Global settings that apply to all tests.
tests	array	Yes	Array of runbook tests to run. At least one test case is required.

Settings object

settings:
  use_temp_working_dir: true    # boolean, default: true
  working_dir: "."              # string, default: current directory
  output_path: "generated"      # string, default: "generated"
  timeout: "5m"                 # duration string, default: "5m"
  parallelizable: true          # boolean, default: true

Field	Type	Default	Description
use_temp_working_dir	boolean	true	Run each test case in its own fresh temporary directory, which is removed when that test case ends. Overrides working_dir when true.
working_dir	string	current directory	Base directory for script execution and file generation. Use . for the runbook’s directory. Any other relative path resolves against the runbook’s directory. Ignored when use_temp_working_dir is true. All test cases share this directory, so files one test case generates or clones are still there for the next.
output_path	string	"generated"	Directory where generated files are written, relative to the working directory.
timeout	string	"5m"	Time limit for each script a block runs and each git command a <GitClone> runs, as a duration such as 500ms, 30s, 5m, or 1h. There is no limit on the test case as a whole. A script assertion is limited to this value or 30 seconds, whichever is lower, and each cleanup action to 30 seconds.
parallelizable	boolean	true	Whether this runbook’s tests can run in parallel with tests from other runbooks. Runbooks are currently tested one at a time, so this only affects order: parallelizable runbooks run first.

Test case object

tests:
  - name: "test-name"           # string, required
    description: "..."          # string, optional
    env: { ... }                # map[string]string, optional
    inputs: { ... }             # map[string]InputValue, optional
    steps: [ ... ]              # array of Step objects, optional
    assertions: [ ... ]         # array of Assertion objects, optional
    cleanup: [ ... ]            # array of CleanupAction objects, optional

Field	Type	Required	Description
name	string	Yes	Unique identifier for the test case.
description	string	No	Human-readable description of what the test validates.
env	map	No	Environment variables to set for all blocks, script assertions and cleanup actions in this test. Keys and values are strings.
inputs	map	No	Variable values for the test. Keys use inputsID.varName format. Values can be literals or fuzz configs.
steps	array	No	Blocks to execute, in the order listed. A block can be listed more than once, and blocks that aren’t listed don’t run. A step whose block matches no block in the runbook fails the test. If empty, every block runs once in document order and is expected to succeed, except pull request blocks, which are expected to skip.
assertions	array	No	Validations to run after all steps complete.
cleanup	array	No	Actions to run after the test’s blocks and assertions, even if one of them fails.

Test case environment

Each test case starts from the environment the test command was started with. Within a test case, a bash block that succeeds or warns passes the variables it exports and the directory it cds into on to the blocks after it, as in the app, and an auth block adds its credentials. The test case’s env is applied on top of all of that. Blocks, script assertions, and cleanup actions all see this environment.

None of it carries over into the next test case. With use_temp_working_dir, each test case also gets its own empty working directory.

Step object

steps:
  - block: "block-id"           # string, required
    expect: success             # ExpectedStatus, default: "success"
    env_prefix: "PREFIX_"       # string, optional
    missing_outputs: ["..."]    # array of strings, optional
    error_contains: "..."       # string, optional
    assertions: [ ... ]         # array of Assertion objects, optional
    set_outputs: { ... }        # map of strings, optional (Iframe blocks only)

Field	Type	Required	Description
block	string	Yes	ID of the block to execute.
expect	string	No	Expected execution status. Default: success. See Expected status values.
env_prefix	string	No	Prefix for the environment variables an auth block reads its credentials from. See Testing AwsAuth blocks.
missing_outputs	array	No	Expected missing outputs (used with blocked status to verify why a block is blocked). Write each one as templates reference it, with hyphens in the block ID turned into underscores: outputs.create_account.account_id for block create-account. Don’t list an output the block reads only behind a hasKey guard: once its block has run, the app doesn’t wait for it, but the test would still report the block as blocked.
error_contains	string	No	Expected error message substring (used with config_error status).
assertions	array	No	Assertions to run immediately after this step completes.
set_outputs	map	No	For an <Iframe> block, the outputs its page would set. Each name must be in the block’s outputs prop, and each value must be a string. See Testing Iframe blocks.

Expected status values

Status	Description
success	Block completed successfully (exit code 0).
fail	Block failed (any exit code other than 0 or 2).
warn	Block completed with warning (exit code 2).
blocked	Block is blocked: an auth block it depends on hasn’t run or was skipped, a block its dependsOn names hasn’t finished with a success or a warning, or the outputs listed in missing_outputs don’t exist yet. The block doesn’t run.
skip	Don’t run this block, for example an <AwsAuth> in a test that has no AWS credentials. Pull request blocks can only be tested with skip or blocked.
config_error	Block has a configuration error, such as a missing prop.

A script that runs past the timeout setting gets the status timeout. No expect value matches it, so the step fails.

Block IDs

Each step references a block by the value of its id prop. For example, <Check id="verify-install" ... /> is block: verify-install.

Assertion object

All assertions have a type field that determines which other fields are required.

assertions:
  - type: "assertion_type"      # AssertionType, required
    # Additional fields depend on the assertion type

Assertion type	Required fields	Optional fields	Description
file_exists	path	none	Check that a file exists at the given path.
file_not_exists	path	none	Check that a file does not exist.
file_contains	path, contains	none	Check that a file contains a substring.
file_not_contains	path, contains	none	Check that a file does not contain a substring.
file_matches	path, pattern	none	Check that file content matches a regex pattern.
file_equals	path, value	none	Check that file content exactly equals a value.
dir_exists	path	none	Check that a directory exists.
dir_not_exists	path	none	Check that a directory does not exist.
output_equals	block, output, value	none	Check that a block output equals a value.
output_matches	block, output, pattern	none	Check that a block output matches a regex pattern.
output_exists	block, output	none	Check that a block output exists (is not empty).
files_generated	block	min_count	Check that a block generated files.
script	command	none	Run a shell command. Exit code 0 passes.

files_generated counts only the files the named block wrote during the test case: the files a <Template> rendered, the file a <TemplateInline> with generateFile wrote, or the files a <Command> or <Check> wrote to $GENERATED_FILES. Templates with target="worktree" count too, though they write into the cloned repository. Files that other blocks or earlier test cases left in the output directory don’t count.

Assertion fields

Field	Type	Description
type	string	The assertion type. Required for all assertions.
path	string	File or directory path, relative to the output directory (<working_dir>/<output_path>, which is generated/ in the working directory by default). An absolute path is used as is.
contains	string	Substring to search for in file content.
pattern	string	Regular expression pattern.
value	string	Expected exact value.
block	string	Block ID for output assertions or files_generated.
output	string	Output name to check.
min_count	integer	Minimum number of files generated (for files_generated). Default: 1.
command	string	Shell command for a script assertion. It runs in bash from the output directory.

Cleanup action object

cleanup:
  - command: "rm -rf /tmp/test" # string, optional (inline command)
    path: "cleanup/teardown.sh" # string, optional (script file path)

Field	Type	Description
command	string	Inline shell command to execute.
path	string	Path to a script file to execute, relative to the runbook’s directory.

When both are set, command runs and path is ignored.

Input values

An input value is a literal or a fuzz configuration.

Literal values

inputs:
  project.Name: "my-project"    # string literal
  config.Port: 8080             # integer literal
  config.Enabled: true          # boolean literal

Fuzz values

A fuzz value is generated at random within the given constraints, anew on each test run.

inputs:
  project.Name:
    fuzz:
      type: string              # FuzzType, required
      # Additional fields depend on the fuzz type

Variables a test doesn’t set

A test only needs to set the variables it cares about. Every other variable of an <Inputs> or <Template> block gets the value the app’s form starts it with, so a field a user can leave untouched in the app can be left out of the test too:

• Its default, if it has one.
• Otherwise, the value its field shows before anyone touches it: false for a bool (an unchecked checkbox), and for a list with an x-schema (a fixed row of fields), one element per schema key, false for a bool element and "" for any other.
• Any other variable without a default stays unset, as it does in the form, so set it in the test if a block uses it.

When a <Template> imports a block with inputsId and both define a variable, the app shows that variable read-only with the imported block’s value, so the test gives it that value too. For example, setting config.Environment also sets vpc.Environment for <Template id="vpc" inputsId="config" />. With several inputsIds, the last one that has a value wins.

A value the test sets always wins over one filled in this way.

Fuzz configuration

The fuzz object specifies how to generate random values.

Field	Type	Applies to	Description
type	string	All	Required. One of the fuzz types.
length	integer	string	Exact length of the generated string, not counting prefix and suffix. Overrides minLength/maxLength. 0 gives an empty string.
minLength	integer	string, list, map	Minimum length of the string, of each list item, or of each map value (without schema). Default: 8 for strings, 5 for list items and map values, or maxLength when only a smaller maxLength is set.
maxLength	integer	string, list, map	Maximum length. Must be at least minLength. Default: minLength + 10 for strings, minLength + 7 for list items and map values.
prefix	string	string	Prefix to prepend to the generated string.
suffix	string	string	Suffix to append to the generated string.
includeSpaces	boolean	string	Include space characters. Default: false.
includeSpecialChars	boolean	string	Include special characters. Default: false.
min	number	int, float	Minimum numeric value. Default: 0, or max - 100 when only a zero or negative max is set.
max	number	int, float	Maximum numeric value. Must be at least min; equal to min for a fixed value. Default: min + 100. For int, fractional bounds narrow to the integers between them, and the test case fails if there are none.
options	array	enum	Array of valid enum options. Required for enum type.
domain	string	email, url	Domain to use. Default: random from example.com, test.org, etc.
minDate	string	date, timestamp	Minimum date (supports RFC3339, YYYY-MM-DD). Default: 365 days ago.
maxDate	string	date, timestamp	Maximum date. Default: now.
format	string	date, timestamp	Output format as a Go reference layout, evaluated in UTC. Supported tokens: 2006 (year), 01 (month), 02 (day), 15 (hour), 04 (minute), 05 (second), and the zones Z07:00 and Z0700 (written as Z), -07:00 (+00:00), -0700 (+0000) and MST (UTC). Other text is copied as is. Default: 2006-01-02 for date, RFC3339 for timestamp.
wordCount	integer	words	Exact number of words. Overrides minWordCount/maxWordCount. 0 gives an empty string.
minWordCount	integer	words	Minimum word count. Default: 2, or maxWordCount when only a smaller maxWordCount is set.
maxWordCount	integer	words	Maximum word count. Must be at least minWordCount. Default: minWordCount + 3.
count	integer	list, map	Exact number of items/entries. Overrides minCount/maxCount. 0 gives an empty list or map.
minCount	integer	list, map	Minimum items/entries. Default: 2, or maxCount when only a smaller maxCount is set.
maxCount	integer	list, map	Maximum items/entries. Must be at least minCount. Default: minCount + 3 for lists, minCount + 2 for maps.
schema	array	map	Field names for nested map values (generates map[string]map[string]string).

A maximum below its minimum fails the test case with an error such as fuzz string: maxLength (3) is less than minLength (5). Setting a minimum and maximum to the same value gives exactly that value. A field left empty (max: or max: ~) counts as unset.

Fuzz types

Type	Output	Description
string	string	Alphanumeric string with optional spaces and special characters.
int	integer	Random integer within min/max range.
float	float	Random float within min/max range.
bool	boolean	Random true or false.
enum	string	Random selection from options array.
email	string	Valid email address (e.g., abc123@example.com).
url	string	Valid URL (e.g., https://example.com/path).
uuid	string	UUID v4 (e.g., 550e8400-e29b-41d4-a716-446655440000).
date	string	Date string. Default format: YYYY-MM-DD.
timestamp	string	Timestamp string. Default format: RFC3339.
words	string	Space-separated random words.
list	string	JSON array of strings (e.g., ["abc", "def"]).
map	string or map	JSON object or nested map depending on schema.

Complete example

version: 1

settings:
  use_temp_working_dir: true
  output_path: generated
  timeout: 10m
  parallelizable: true

tests:
  - name: happy-path
    description: Standard successful execution with all features

    env:
      LOG_LEVEL: debug
      RUNBOOK_DRY_RUN: "false"

    inputs:
      # Literal values
      project.Name: "test-project"
      config.Port: 8080

      # Fuzz values
      project.Description:
        fuzz: { type: words, minWordCount: 3, maxWordCount: 6 }
      user.Email:
        fuzz: { type: email, domain: "test.example.com" }
      resource.ID:
        fuzz: { type: uuid }
      config.Region:
        fuzz: { type: enum, options: ["us-east-1", "us-west-2", "eu-west-1"] }

    steps:
      - block: validate-inputs
        expect: success

      - block: create-resources
        expect: success
        assertions:
          - type: output_exists
            block: create-resources
            output: resource_id

      - block: generate-config
        expect: success

    assertions:
      - type: file_exists
        path: config.json

      - type: file_contains
        path: config.json
        contains: "test-project"

      - type: file_matches
        path: config.json
        pattern: '"version":\s*"\d+\.\d+\.\d+"'

      - type: output_matches
        block: create-resources
        output: resource_id
        pattern: "^[a-f0-9]{12}$"

    cleanup:
      - command: rm -rf /tmp/test-resources

  - name: missing-dependency
    description: Test that blocks are properly blocked

    steps:
      - block: create-resources
        expect: blocked
        missing_outputs:
          - outputs.validate_inputs.validated

      - block: validate-inputs
        expect: success

      - block: create-resources
        expect: success

Cleanup

Cleanup actions run after the test’s blocks and assertions, even if a block or assertion fails:

cleanup:
  # Inline command
  - command: rm -rf /tmp/test-resources

  # Script file
  - path: cleanup/teardown.sh

Both kinds of action run in bash from the output directory (<working_dir>/<output_path>), which is created first if no block has generated files yet. A path script is read relative to the runbook’s directory. They see the test case environment, and each is limited to 30 seconds. A failing cleanup action doesn’t fail the test and doesn’t stop the remaining actions, but it is printed as a warning.

A test that fails before any block runs skips cleanup, since nothing has run yet. That happens when the runbook has a block type the runner doesn’t know, when the test’s inputs don’t validate, and when a step names a block the runbook doesn’t have.

Testing AwsAuth blocks

For an <AwsAuth> block, the test runner looks for AWS credentials in environment variables. It never contacts AWS. It checks these sources in order:

1. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, with AWS_SESSION_TOKEN and AWS_REGION if they are set
2. AWS_PROFILE
3. AWS_ROLE_ARN together with AWS_WEB_IDENTITY_TOKEN_FILE

Static keys go into the test case environment, so every later block sees them. A profile, or a role and token file, is passed only to blocks that reference the auth block with awsAuthId. If none of the three is set, the step is skipped and blocks that reference the auth block are blocked.

Using a test prefix

To keep test credentials apart from your local AWS credentials, set env_prefix on the test step:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: aws-auth
        env_prefix: RUNBOOKS_TEST_
        expect: success

The runner then reads:

• RUNBOOKS_TEST_AWS_ACCESS_KEY_ID
• RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY
• RUNBOOKS_TEST_AWS_SESSION_TOKEN
• RUNBOOKS_TEST_AWS_REGION

If the prefixed key pair isn’t set, it falls back to the unprefixed variables. The prefix applies only to these four. AWS_PROFILE, AWS_ROLE_ARN, and AWS_WEB_IDENTITY_TOKEN_FILE are read without it.

CI example

In GitHub Actions:

.github/workflows/test.yml

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
    aws-region: us-west-2

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: ${{ env.AWS_ACCESS_KEY_ID }}
    RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
    RUNBOOKS_TEST_AWS_SESSION_TOKEN: ${{ env.AWS_SESSION_TOKEN }}
    RUNBOOKS_TEST_AWS_REGION: ${{ env.AWS_REGION }}

Testing GoogleAuth blocks

For a <GoogleAuth> block, the test runner looks for Google Cloud credentials in environment variables and adds them to the test case environment. As with AwsAuth, it never contacts Google, so a test shows that the block is wired up and says nothing about whether the credential is valid.

It checks these credential variables in order:

• GOOGLE_APPLICATION_CREDENTIALS (path to a credentials JSON file, either a service account key or an authorized_user document)
• GOOGLE_CREDENTIALS (inline credentials JSON)
• GOOGLE_OAUTH_ACCESS_TOKEN
• CLOUDSDK_AUTH_ACCESS_TOKEN

If none is set, it falls back to gcloud’s application_default_credentials.json, the block’s 'adc' detection source. If that is missing too, the step is skipped and every block that references it with googleAuthId is blocked.

It also reads these variables, none of which counts as a credential:

• Project: CLOUDSDK_CORE_PROJECT, GOOGLE_CLOUD_PROJECT, GOOGLE_PROJECT, GCLOUD_PROJECT
• Region: CLOUDSDK_COMPUTE_REGION, GOOGLE_CLOUD_REGION
• Zone: CLOUDSDK_COMPUTE_ZONE

When none of those is set, the block’s own project, defaultRegion, and defaultZone props are used.

On success the runner writes the same variables the app’s session environment writes, so scripts see every name that gcloud, the client libraries, and the OpenTofu google provider each look for: GOOGLE_CLOUD_PROJECT / CLOUDSDK_CORE_PROJECT / GOOGLE_PROJECT, GOOGLE_CLOUD_REGION / CLOUDSDK_COMPUTE_REGION / GOOGLE_REGION, and CLOUDSDK_COMPUTE_ZONE / GOOGLE_ZONE. A credentials file also sets CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE, so gcloud uses that file and ignores any account from gcloud auth login. An access token clears it.

Using a test prefix

As with AwsAuth, set env_prefix on the test step to keep test credentials apart from your local ones:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: google-auth
        env_prefix: RUNBOOKS_TEST_
        expect: success

The runner then checks:

• RUNBOOKS_TEST_GOOGLE_APPLICATION_CREDENTIALS
• RUNBOOKS_TEST_GOOGLE_CREDENTIALS
• RUNBOOKS_TEST_GOOGLE_OAUTH_ACCESS_TOKEN
• RUNBOOKS_TEST_CLOUDSDK_AUTH_ACCESS_TOKEN
• RUNBOOKS_TEST_CLOUDSDK_CORE_PROJECT (and the other project/region/zone names above, prefixed)

If no prefixed credential is found, it falls back to the unprefixed variables.

CI example

.github/workflows/test.yml

- name: Authenticate to Google Cloud
  uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: projects/123/locations/global/workloadIdentityPools/ci/providers/github
    service_account: runbooks-ci@my-project.iam.gserviceaccount.com

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_TEST_GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.auth.outputs.credentials_file_path }}
    RUNBOOKS_TEST_CLOUDSDK_CORE_PROJECT: my-project
    RUNBOOKS_TEST_CLOUDSDK_COMPUTE_REGION: us-central1

Testing GitHubAuth blocks

For a <GitHubAuth> block, the test runner looks for a GitHub token in these environment variables, in order:

1. RUNBOOKS_GITHUB_TOKEN (recommended for tests)
2. GITHUB_TOKEN
3. GH_TOKEN

Using a test prefix

With env_prefix on the step, the runner reads the prefixed names:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: gh-auth
        env_prefix: CI_
        expect: success

This step reads CI_GITHUB_TOKEN and CI_GH_TOKEN.

CI example

.github/workflows/test.yml

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Testing GitLabAuth and GitAuth blocks

For a <GitLabAuth> block, the test runner looks for a GitLab token in these environment variables, in order:

1. GITLAB_TOKEN
2. GITLAB_ACCESS_TOKEN
3. OAUTH_TOKEN

With env_prefix on the step, it checks the prefixed names instead (CI_GITLAB_TOKEN, CI_GITLAB_ACCESS_TOKEN, CI_OAUTH_TOKEN).

On success, the blocks after it see GITLAB_TOKEN and GITLAB_HOST. As in the app, a token from the environment belongs to one GitLab host: the one GITLAB_HOST (or GITLAB_URI, GL_HOST) names, or gitlab.com if none is set. A block that pins another host with instanceUrl or host skips unless GITLAB_HOST names that host too, and a block whose pinned value isn’t a valid URL or host always skips.

The test runner never sends the token anywhere, so a fake one is enough to test that the block authenticates:

runbook_test.yml

tests:
  - name: detects-env-token
    env:
      GITLAB_TOKEN: fake-token
    steps:
      - block: gitlab-auth
        expect: success

A <GitAuth> block is tested like <GitHubAuth> or <GitLabAuth>, depending on its provider prop (github by default).

Cloning with a token

A <GitClone> that references an auth block with gitAuthId or githubAuthId clones an https URL with that block’s token when the URL is on the host the block authenticated to. That includes GitHub Enterprise Server and self-managed GitLab hosts. The username is oauth2 for GitLab and x-access-token for GitHub.

Without a reference, the runner recognizes github.com, *.ghe.com, and gitlab.com URLs and uses that provider’s token from the test case environment: GITHUB_TOKEN or GH_TOKEN for GitHub, GITLAB_TOKEN for GitLab. A token already set where the tests run counts too, so to clone without one, blank it in the test case’s env:

tests:
  - name: anonymous-clone
    env:
      GITHUB_TOKEN: ""
      GH_TOKEN: ""

Testing pull request blocks

<GitPullRequest>, <GitHubPullRequest>, and <GitLabMergeRequest> blocks never run in tests, since they would push a branch and open a real pull request. A step that names one must use expect: skip, or expect: blocked when the auth block it references hasn’t run or was skipped. A test with no steps expects them to skip.

runbook_test.yml

tests:
  - name: skip-without-credentials
    steps:
      - block: gh-auth
        expect: skip
      - block: create-pr
        expect: skip

Testing Iframe blocks

Tests can’t run the page an <Iframe> block shows, so a step for the block stands in for the page: its set_outputs become the block’s outputs, as if the page had set them. Only an <Iframe> with an id can be a step.

runbook_test.yml

tests:
  - name: deploy-to-picked-region
    steps:
      - block: region-picker
        set_outputs:
          region: eu-west-1
      - block: deploy
        expect: success

A step whose set_outputs names an output the block’s outputs prop doesn’t list fails, just as the app refuses that output from the page. Without set_outputs, the step succeeds and sets nothing, so blocks that read the block’s outputs stay blocked. A test with no steps runs each <Iframe> with an id that way.

Running tests

The runner executes each test case’s blocks in a temporary directory by default, captures script output and generated files, and checks them against the test’s assertions. For a failed test it prints which block failed and why.

It exits with status 0 if every test passed. It exits with 1 if any test failed, a runbook_test.yml couldn’t be loaded, or no runbook has the test case named with --test.

Single runbook

# Run all tests for a runbook
runbooks-cli test ./my-runbook

# Run a specific test case
runbooks-cli test ./my-runbook --test happy-path

# Verbose output
runbooks-cli test ./my-runbook -v

Multiple runbooks

End the path with /... to test every runbook under a directory:

# Test all runbooks in a directory tree
runbooks-cli test ./runbooks/...

Runbooks are tested one at a time. The --max-parallel flag is accepted but currently has no effect.

CI output

To write a JUnit XML report:

runbooks-cli test ./runbooks/... --output junit --output-file results.xml

The file is written before the command exits, including when tests fail. If it can’t be written, the CLI prints the error and writes the report to stdout instead.

Out-of-order testing

To run blocks in a different order than the runbook defines them, list them under steps. Steps run in the order listed, and a block listed more than once runs once per step, each time with that step’s expectation.

A block added to the runbook later is not tested until you add it to steps. Leave steps empty where you can, so new blocks are covered without a change to the test.

tests:
  - name: out-of-order
    description: Test dependency enforcement

    steps:
      # This should be blocked because create-account hasn't run
      - block: create-resources
        expect: blocked
        missing_outputs:
          - outputs.create_account.account_id

      # Run the dependency
      - block: create-account
        expect: success

      # Now this should succeed
      - block: create-resources
        expect: success

Example workflow

These jobs run in your own repository, which has no copy of the test runner. So each one checks out the Runbooks repository into .runbooks, builds the runner there, and then runs it against the runbooks in your repository’s runbooks directory. Point the runner at that directory rather than ./..., which would also find the test runbooks in .runbooks.

GitHub Actions

.github/workflows/test-runbooks.yml

name: Test Runbooks

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      # The test runner is built from the Runbooks repository.
      # Set `ref` to a release tag to pin the version.
      - uses: actions/checkout@v4
        with:
          repository: gruntwork-io/runbooks
          path: .runbooks

      - uses: jdx/mise-action@v2
        with:
          working_directory: .runbooks

      - name: Build the test runner
        working-directory: .runbooks
        run: |
          bun install --frozen-lockfile
          just compile-test-cli

      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Run runbook tests
        run: .runbooks/resources/bin/runbooks-test test ./runbooks/... --output junit --output-file results.xml
        env:
          RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: ${{ env.AWS_ACCESS_KEY_ID }}
          RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
          RUNBOOKS_TEST_AWS_SESSION_TOKEN: ${{ env.AWS_SESSION_TOKEN }}
          RUNBOOKS_TEST_AWS_REGION: ${{ vars.AWS_REGION }}

      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: results.xml

GitLab CI

.gitlab-ci.yml

test-runbooks:
  stage: test
  variables:
    RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
    RUNBOOKS_TEST_AWS_REGION: $AWS_REGION
  before_script:
    # The test runner is built from the Runbooks repository.
    # Add `--branch <release tag>` to pin the version.
    - git clone --depth 1 https://github.com/gruntwork-io/runbooks.git .runbooks
    - curl -fsSL https://mise.jdx.dev/install.sh | sh
    - export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"
    - cd .runbooks
    - mise install
    - bun install --frozen-lockfile
    - just compile-test-cli
    - cd "$CI_PROJECT_DIR"
  script:
    - .runbooks/resources/bin/runbooks-test test ./runbooks/... --output junit --output-file results.xml
  artifacts:
    reports:
      junit: results.xml

Testing with external dependencies

For runbooks that call external services such as GitHub or AWS, test in tiers. Each tier covers more than the one before it and is slower and harder to set up, as in the test pyramid.

Tier 1: template validation

Tier 1 checks that templates generate files and makes no external calls.

tests:
  - name: template-validation
    description: Validate template generation

    inputs:
      config.ProjectName: "test-project"
      config.Region: "us-east-1"

    steps:
      - block: generate-config  # Template block
        expect: success
      - block: deploy-resources  # Skip external calls
        expect: skip

    assertions:
      - type: files_generated
        block: generate-config
        min_count: 1

Tier 2: dry-run mode

Tier 2 runs scripts with their external calls switched off. The script checks an environment variable, and the test sets it. RUNBOOK_DRY_RUN is a convention, and any variable name works.

In the Command or Check script:

#!/bin/bash
set -e

# Dry-run support
DRY_RUN="${RUNBOOK_DRY_RUN:-false}"

if [[ "$DRY_RUN" == "true" ]]; then
    echo "[DRY-RUN] Would create PR in $GITHUB_ORG/$REPO_NAME"
    echo "[DRY-RUN] gh pr create --title '$TITLE'"
    exit 0
fi

# Real execution
gh pr create --title "$TITLE" --body "$BODY"

In the test configuration:

tests:
  - name: dry-run-flow
    description: Test full flow without real API calls

    env:
      RUNBOOK_DRY_RUN: "true"

    steps:
      - block: generate-config
        expect: success
      - block: create-pr
        expect: success  # Succeeds in dry-run mode

Tier 3: integration tests

Tier 3 runs against real services. Set credentials as environment variables in CI, and run these tests manually or on a schedule:

tests:
  - name: integration-test
    description: Full integration (requires credentials)

    env:
      # Credentials set in the CI environment are available to the test,
      # so they don't need to be set here.

    steps:
      - block: deploy-resources
        expect: success

    cleanup:
      - path: scripts/cleanup-test-resources.sh

Best practices

• Start with one test case and no steps, so every block runs in document order. Add inputs, steps, and assertions from there.
• Cover the failure paths too: expect: fail for a script that should exit non-zero, expect: blocked with missing_outputs for a block whose dependency hasn’t run, and expect: config_error with error_contains for a misconfigured block.