Testing Runbooks

A runbook test runs the runbook’s blocks from the command line and checks their results. You describe the test cases in a runbook_test.yml file next to runbook.mdx, then run runbooks-cli test /path/to/runbook, or runbooks-cli test ./... for every runbook under the current directory. Tests run locally or in CI.

The test runner is a standalone CLI in the Runbooks repository. From a checkout, just compile-test-cli builds it into a single binary at resources/bin/runbooks-test, which runs without Node.js or Bun. You can also run it from source with bun cli/index.ts test. The examples below use runbooks-cli as shorthand for either invocation.

Quick start

1. Create a runbook_test.yml file next to your runbook.mdx. This minimal configuration runs every block in document order and expects each one to succeed:

version: 1

tests:
  - name: happy-path

Add inputs, steps, and assertions as you need them. For a fuller example to start from, see testdata/sample-runbooks/my-first-runbook/runbook_test.yml in the Runbooks repository.

2. Run the tests:

runbooks-cli test ./my-runbook

Test configuration

A fuller runbook_test.yml:

version: 1

settings:
  # use_temp_working_dir: true  # Default: use isolated temp directory
  # working_dir: .              # Alternative: use runbook's directory
  # output_path: generated      # Where to write generated files (relative to working_dir)
  timeout: 5m            # Limit for each script and git command
  parallelizable: true   # Can run in parallel with other runbooks

tests:
  - name: happy-path
    description: Standard successful execution

    inputs:
      project.Name: "test-project"
      project.Language: "python"

    steps:
      - block: check-requirements
        expect: success

      - block: create-project
        expect: success

    assertions:
      - type: file_exists
        path: README.md

      - type: file_contains
        path: README.md
        contains: "test-project"

    cleanup:
      - command: rm -rf /tmp/test-resources

Top-level structure

version: 1              # Required. Must be 1.
settings: { ... }       # Optional. Global settings for all tests.
tests: [ ... ]          # Required. Array of test cases (minimum 1).

Field	Type	Required	Description
version	integer	Yes	Configuration version. Must be 1.
settings	object	No	Global settings that apply to all tests.
tests	array	Yes	Array of runbook tests to run. At least one test case is required.

Settings object

settings:
  use_temp_working_dir: true    # boolean, default: true
  working_dir: "."              # string, default: current directory
  output_path: "generated"      # string, default: "generated"
  timeout: "5m"                 # duration string, default: "5m"
  parallelizable: true          # boolean, default: true

Field	Type	Default	Description
use_temp_working_dir	boolean	true	Run each test case in its own fresh temporary directory, which is removed when that test case ends. Overrides working_dir when true.
working_dir	string	current directory	Base directory for script execution and file generation. Use . for the runbook’s directory. Any other relative path resolves against the runbook’s directory. Ignored when use_temp_working_dir is true. All test cases share this directory, so files one test case generates or clones are still there for the next.
output_path	string	"generated"	Directory where generated files are written, relative to the working directory.
timeout	string	"5m"	Time limit for each script a block runs and each git command a <GitClone> runs, as a duration such as 500ms, 30s, 5m, or 1h. Tests ignore a block’s timeoutMs, so this limit applies to every script. There is no limit on the test case as a whole. A script assertion is limited to this value or 30 seconds, whichever is lower, and each cleanup action to 30 seconds.
parallelizable	boolean	true	Whether this runbook’s tests can run in parallel with tests from other runbooks. Runbooks are currently tested one at a time, so this only affects order: parallelizable runbooks run first.

Test case object

tests:
  - name: "test-name"           # string, required
    description: "..."          # string, optional
    env: { ... }                # map[string]string, optional
    inputs: { ... }             # map[string]InputValue, optional
    steps: [ ... ]              # array of Step objects, optional
    assertions: [ ... ]         # array of Assertion objects, optional
    cleanup: [ ... ]            # array of CleanupAction objects, optional

Field	Type	Required	Description
name	string	Yes	Unique identifier for the test case.
description	string	No	Human-readable description of what the test validates.
env	map	No	Environment variables to set for all blocks, script assertions and cleanup actions in this test. Keys and values are strings.
inputs	map	No	Variable values for the test. Keys use inputsID.varName format. Values can be literals or fuzz configs.
steps	array	No	Blocks to execute, in the order listed. A block can be listed more than once, and blocks that aren’t listed don’t run. A step whose block matches no block in the runbook fails the test. If empty, every block runs once in document order and is expected to succeed, except pull request blocks, which are expected to skip.
assertions	array	No	Validations to run after all steps complete.
cleanup	array	No	Actions to run after the test’s blocks and assertions, even if one of them fails.

Test case environment

Each test case starts from the environment the test command was started with. Within a test case, a bash block that succeeds or warns passes the variables it exports and the directory it cds into on to the blocks after it, as in the app, and an auth block adds its credentials. The test case’s env is applied on top of all of that. Blocks, script assertions, and cleanup actions all see this environment.

None of it carries over into the next test case. With use_temp_working_dir, each test case also gets its own empty working directory.

Step object

steps:
  - block: "block-id"           # string, required
    expect: success             # ExpectedStatus, default: "success"
    env_prefix: "PREFIX_"       # string, optional
    missing_outputs: ["..."]    # array of strings, optional
    error_contains: "..."       # string, optional
    assertions: [ ... ]         # array of Assertion objects, optional
    set_outputs: { ... }        # map of strings, optional (Iframe blocks only)

Field	Type	Required	Description
block	string	Yes	ID of the block to execute.
expect	string	No	Expected execution status. Default: success. See Expected status values.
env_prefix	string	No	Prefix for the environment variables an auth block reads its credentials from. See Testing AwsAuth blocks.
missing_outputs	array	No	Expected missing outputs (used with blocked status to verify why a block is blocked). Write each one as templates reference it, with hyphens in the block ID turned into underscores: outputs.create_account.account_id for block create-account. Don’t list an output the block reads only behind a hasKey guard: once its block has run, the app doesn’t wait for it, but the test would still report the block as blocked.
error_contains	string	No	Expected error message substring (used with config_error status).
assertions	array	No	Assertions to run immediately after this step completes.
set_outputs	map	No	For an <Iframe> block, the outputs its page would set. Each name must be in the block’s outputs prop, and each value must be a string. See Testing Iframe blocks.

Expected status values

Status	Description
success	Block completed successfully (exit code 0).
fail	Block failed (any exit code other than 0 or 2).
warn	Block completed with warning (exit code 2).
blocked	Block is blocked: an auth block it depends on hasn’t run or was skipped, a block its dependsOn names hasn’t finished with a success or a warning, or the outputs listed in missing_outputs don’t exist yet. The block doesn’t run.
skip	Don’t run this block, for example an <AwsAuth> in a test that has no AWS credentials. Pull request blocks can only be tested with skip or blocked.
config_error	Block has a configuration error, such as a missing prop.

A script that runs past the timeout setting gets the status timeout. No expect value matches it, so the step fails.

Block IDs

Each step references a block by the value of its id prop. For example, <Check id="verify-install" ... /> is block: verify-install.

Assertion object

All assertions have a type field that determines which other fields are required.

assertions:
  - type: "assertion_type"      # AssertionType, required
    # Additional fields depend on the assertion type

Assertion type	Required fields	Optional fields	Description
file_exists	path	none	Check that a file exists at the given path.
file_not_exists	path	none	Check that a file does not exist.
file_contains	path, contains	none	Check that a file contains a substring.
file_not_contains	path, contains	none	Check that a file does not contain a substring.
file_matches	path, pattern	none	Check that file content matches a regex pattern.
file_equals	path, value	none	Check that file content exactly equals a value.
dir_exists	path	none	Check that a directory exists.
dir_not_exists	path	none	Check that a directory does not exist.
output_equals	block, output, value	none	Check that a block output equals a value.
output_matches	block, output, pattern	none	Check that a block output matches a regex pattern.
output_exists	block, output	none	Check that a block output exists (is not empty).
files_generated	block	min_count	Check that a block generated files.
script	command	none	Run a shell command. Exit code 0 passes.

files_generated counts only the files the named block wrote during the test case: the files a <Template> rendered, the file a <TemplateInline> with generateFile wrote, or the files a <Command> or <Check> wrote to $GENERATED_FILES. Templates with target="worktree" count too, though they write into the cloned repository. Files that other blocks or earlier test cases left in the output directory don’t count.

Assertion fields

Field	Type	Description
type	string	The assertion type. Required for all assertions.
path	string	File or directory path, relative to the output directory (<working_dir>/<output_path>, which is generated/ in the working directory by default). An absolute path is used as is.
contains	string	Substring to search for in file content.
pattern	string	Regular expression pattern.
value	string	Expected exact value.
block	string	Block ID for output assertions or files_generated.
output	string	Output name to check.
min_count	integer	Minimum number of files generated (for files_generated). Default: 1.
command	string	Shell command for a script assertion. It runs in bash from the output directory.

Cleanup action object

cleanup:
  - command: "rm -rf /tmp/test" # string, optional (inline command)
    path: "cleanup/teardown.sh" # string, optional (script file path)

Field	Type	Description
command	string	Inline shell command to execute.
path	string	Path to a script file to execute, relative to the runbook’s directory.

When both are set, command runs and path is ignored.

Input values

An input value is a literal or a fuzz configuration.

Literal values

inputs:
  project.Name: "my-project"    # string literal
  config.Port: 8080             # integer literal
  config.Enabled: true          # boolean literal

Fuzz values

A fuzz value is generated at random within the given constraints, anew on each test run.

inputs:
  project.Name:
    fuzz:
      type: string              # FuzzType, required
      # Additional fields depend on the fuzz type

Variables a test doesn’t set

A test only needs to set the variables it cares about. Every other variable of an <Inputs> or <Template> block gets the value the app’s form starts it with, with the two exceptions below, so a field a user can leave untouched in the app can be left out of the test too:

• Its default, if it has one.
• Otherwise, the value its field shows before anyone touches it: false for a bool (an unchecked checkbox), and for a list with an x-schema (a fixed row of fields), one element per schema key, false for a bool element and "" for any other.
• Any other variable without a default stays unset, as it does in the form, so set it in the test if a block uses it.

Set the variable in the test in these two cases, where the runner doesn’t start it the way the form does:

• The <Inputs> block gives it a value with prefilledVariables. The runner ignores that prop.
• Its default is built from other variables, such as "{{ .ProjectName }}-state". The app fills in the other variables’ values, but the runner uses the text as written.

When a <Template> imports a block with inputsId and both define a variable, the app shows that variable read-only with the imported block’s value, so the test gives it that value too. For example, setting config.Environment also sets vpc.Environment for <Template id="vpc" inputsId="config" />. With several inputsIds, the last one that has a value wins.

A value the test sets always wins over one filled in this way.

Which blocks get a value

As in the app, a block’s {{ .inputs.Name }} comes only from the blocks it imports. Those are the blocks its inputsId names, merged in order so that a later block’s value wins, and then an <Inputs> nested in a <Check> or <Command>. A <Template> adds its own variables on top of the ones it imports, and a block whose inputsId names a Template gets both.

So when two <Inputs> blocks define Region, setting config.Region changes it only for the blocks that import config. A block that imports nothing has no inputs: if its script reads one, the test fails, and the app shows Configuration Required instead of running it.

Fuzz configuration

The fuzz object specifies how to generate random values.

Field	Type	Applies to	Description
type	string	All	Required. One of the fuzz types.
length	integer	string	Exact length of the generated string, not counting prefix and suffix. Overrides minLength/maxLength. 0 gives an empty string.
minLength	integer	string, list, map	Minimum length of the string, of each list item, or of each map value (without schema). Default: 8 for strings, 5 for list items and map values, or maxLength when only a smaller maxLength is set.
maxLength	integer	string, list, map	Maximum length. Must be at least minLength. Default: minLength + 10 for strings, minLength + 7 for list items and map values.
prefix	string	string	Prefix to prepend to the generated string.
suffix	string	string	Suffix to append to the generated string.
includeSpaces	boolean	string	Include space characters. Default: false.
includeSpecialChars	boolean	string	Include special characters. Default: false.
min	number	int, float	Minimum numeric value. Default: 0, or max - 100 when only a zero or negative max is set.
max	number	int, float	Maximum numeric value. Must be at least min; equal to min for a fixed value. Default: min + 100. For int, fractional bounds narrow to the integers between them, and the test case fails if there are none.
options	array	enum	Array of valid enum options. Required for enum type.
domain	string	email, url	Domain to use. Default: random from example.com, test.org, etc.
minDate	string	date, timestamp	Minimum date (supports RFC3339, YYYY-MM-DD). Default: 365 days ago.
maxDate	string	date, timestamp	Maximum date. Default: now.
format	string	date, timestamp	Output format as a Go reference layout, evaluated in UTC. Supported tokens: 2006 (year), 01 (month), 02 (day), 15 (hour), 04 (minute), 05 (second), and the zones Z07:00 and Z0700 (written as Z), -07:00 (+00:00), -0700 (+0000) and MST (UTC). Other text is copied as is. Default: 2006-01-02 for date, RFC3339 for timestamp.
wordCount	integer	words	Exact number of words. Overrides minWordCount/maxWordCount. 0 gives an empty string.
minWordCount	integer	words	Minimum word count. Default: 2, or maxWordCount when only a smaller maxWordCount is set.
maxWordCount	integer	words	Maximum word count. Must be at least minWordCount. Default: minWordCount + 3.
count	integer	list, map	Exact number of items/entries. Overrides minCount/maxCount. 0 gives an empty list or map.
minCount	integer	list, map	Minimum items/entries. Default: 2, or maxCount when only a smaller maxCount is set.
maxCount	integer	list, map	Maximum items/entries. Must be at least minCount. Default: minCount + 3 for lists, minCount + 2 for maps.
schema	array	map	Field names for nested map values (generates map[string]map[string]string).

A maximum below its minimum fails the test case with an error such as fuzz string: maxLength (3) is less than minLength (5). Setting a minimum and maximum to the same value gives exactly that value. A field left empty (max: or max: ~) counts as unset.

Fuzz types

Type	Output	Description
string	string	Alphanumeric string with optional spaces and special characters.
int	integer	Random integer within min/max range.
float	float	Random float within min/max range.
bool	boolean	Random true or false.
enum	string	Random selection from options array.
email	string	Valid email address (e.g., abc123@example.com).
url	string	Valid URL (e.g., https://example.com/path).
uuid	string	UUID v4 (e.g., 550e8400-e29b-41d4-a716-446655440000).
date	string	Date string. Default format: YYYY-MM-DD.
timestamp	string	Timestamp string. Default format: RFC3339.
words	string	Space-separated random words.
list	string	JSON array of strings (e.g., ["abc", "def"]).
map	string or map	JSON object or nested map depending on schema.

Complete example

version: 1

settings:
  use_temp_working_dir: true
  output_path: generated
  timeout: 10m
  parallelizable: true

tests:
  - name: happy-path
    description: Standard successful execution with all features

    env:
      LOG_LEVEL: debug
      RUNBOOK_DRY_RUN: "false"

    inputs:
      # Literal values
      project.Name: "test-project"
      config.Port: 8080

      # Fuzz values
      project.Description:
        fuzz: { type: words, minWordCount: 3, maxWordCount: 6 }
      user.Email:
        fuzz: { type: email, domain: "test.example.com" }
      resource.ID:
        fuzz: { type: uuid }
      config.Region:
        fuzz: { type: enum, options: ["us-east-1", "us-west-2", "eu-west-1"] }

    steps:
      - block: validate-inputs
        expect: success

      - block: create-resources
        expect: success
        assertions:
          - type: output_exists
            block: create-resources
            output: resource_id

      - block: generate-config
        expect: success

    assertions:
      - type: file_exists
        path: config.json

      - type: file_contains
        path: config.json
        contains: "test-project"

      - type: file_matches
        path: config.json
        pattern: '"version":\s*"\d+\.\d+\.\d+"'

      - type: output_matches
        block: create-resources
        output: resource_id
        pattern: "^[a-f0-9]{12}$"

    cleanup:
      - command: rm -rf /tmp/test-resources

  - name: missing-dependency
    description: Test that blocks are properly blocked

    steps:
      - block: create-resources
        expect: blocked
        missing_outputs:
          - outputs.validate_inputs.validated

      - block: validate-inputs
        expect: success

      - block: create-resources
        expect: success

Cleanup

Cleanup actions run after the test’s blocks and assertions, even if a block or assertion fails:

cleanup:
  # Inline command
  - command: rm -rf /tmp/test-resources

  # Script file
  - path: cleanup/teardown.sh

Both kinds of action run in bash from the output directory (<working_dir>/<output_path>), which is created first if no block has generated files yet. A path script is read relative to the runbook’s directory. They see the test case environment, and each is limited to 30 seconds. A failing cleanup action doesn’t fail the test and doesn’t stop the remaining actions, but it is printed as a warning.

A test that fails before any block runs skips cleanup, since nothing has run yet. That happens when the runbook has a block type the runner doesn’t know, when the test’s inputs don’t validate, and when a step names a block the runbook doesn’t have.

How tests differ from the app

The runner runs your scripts and renders your templates, but not always as the app does. A test can pass while the runbook fails in the app, or the other way around:

• Template syntax. The runner renders scripts, env values, templates and block props with its own renderer. It handles only references such as {{ .inputs.Name }} and {{ .outputs.block_id.name }}, {{ if .inputs.Flag }}...{{ else }}...{{ end }} without nesting, and {{ fromJson .outputs.block_id.name }}. Anything else is left in the result as written, including eq and the other functions, pipes such as | upper, range, with, the raw-string form, and the {{- and -}} markers. A reference inside one of those, such as the output behind a hasKey guard, is still filled in, or reported as undefined.
• Working directory. In the app, scripts start in the session’s directory, never in the runbook’s folder. Under test they start in the working directory: a fresh temporary directory by default, or the runbook’s folder with working_dir: .. So a script that reads a file next to runbook.mdx can pass with working_dir: . and fail in the app.
• Script paths. The app refuses a path with a .. segment, such as ../shared/setup.sh, and shows an error in place of the block. The runner reads the file anyway.
• Timeouts. A block’s timeoutMs doesn’t apply. Every script gets the timeout setting, 5 minutes by default.
• A Command that exits 2. The runner reports it as warn, and the blocks whose dependsOn names it can run. In the app, the Command shows as failed and those blocks keep waiting. A <Check> that exits 2 lets them run in both.
• Auth blocks. The runner ignores detectCredentials. It reads the environment variables listed in each auth block’s section below, and for <GoogleAuth> gcloud’s ADC file, whatever the prop says. A block with detectCredentials={false} still succeeds when they are set, a { block: ... } source is never used, and a prefix in an { env: { prefix } } source has no effect, since only the step’s env_prefix sets one. The runner also checks only that an auth block has an id. Configuration errors that the app reports for an auth block, such as two { block } sources, don’t appear, and an expect: config_error step for one fails.
• Template blocks. A <Template> renders each file with the runner’s renderer. File and directory names aren’t rendered, files and directories whose names start with . aren’t written, and boilerplate.yml is read only for its variables, so settings such as skip_files are ignored.
• TemplateInline blocks. The runner reports a <TemplateInline> without outputPath as a configuration error. In the app, outputPath defaults to template.txt.

Testing AwsAuth blocks

For an <AwsAuth> block, the test runner looks for AWS credentials in environment variables. It never contacts AWS. It checks these sources in order:

1. AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, with AWS_SESSION_TOKEN and AWS_REGION if they are set
2. AWS_PROFILE
3. AWS_ROLE_ARN together with AWS_WEB_IDENTITY_TOKEN_FILE

Static keys go into the test case environment, so every later block sees them. A profile, or a role and token file, is passed only to blocks that reference the auth block with awsAuthId. If none of the three is set, the step is skipped and blocks that reference the auth block are blocked.

The runner sets AWS_REGION only from the AWS_REGION variable. The app falls back to AWS_DEFAULT_REGION and then to the block’s defaultRegion, but the runner doesn’t, so under test a script that reads $AWS_REGION gets nothing when AWS_REGION isn’t set.

Using a test prefix

To keep test credentials apart from your local AWS credentials, set env_prefix on the test step:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: aws-auth
        env_prefix: RUNBOOKS_TEST_
        expect: success

The runner then reads:

• RUNBOOKS_TEST_AWS_ACCESS_KEY_ID
• RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY
• RUNBOOKS_TEST_AWS_SESSION_TOKEN
• RUNBOOKS_TEST_AWS_REGION

If the prefixed key pair isn’t set, it falls back to the unprefixed variables. The prefix applies only to these four. AWS_PROFILE, AWS_ROLE_ARN, and AWS_WEB_IDENTITY_TOKEN_FILE are read without it. A prefixed source in the app never falls back like this, so a test whose prefixed variables are missing can still pass, using whatever AWS credentials the machine running it has.

CI example

In GitHub Actions:

.github/workflows/test.yml

- name: Configure AWS credentials
  uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789012:role/GitHubActionsRole
    aws-region: us-west-2

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: ${{ env.AWS_ACCESS_KEY_ID }}
    RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
    RUNBOOKS_TEST_AWS_SESSION_TOKEN: ${{ env.AWS_SESSION_TOKEN }}
    RUNBOOKS_TEST_AWS_REGION: ${{ env.AWS_REGION }}

Testing GoogleAuth blocks

For a <GoogleAuth> block, the test runner looks for Google Cloud credentials in environment variables and adds them to the test case environment. As with AwsAuth, it never contacts Google, so a test shows that the block is wired up and says nothing about whether the credential is valid.

It checks these credential variables in order:

• GOOGLE_APPLICATION_CREDENTIALS (path to a credentials JSON file, either a service account key or an authorized_user document)
• GOOGLE_CREDENTIALS (inline credentials JSON)
• GOOGLE_OAUTH_ACCESS_TOKEN
• CLOUDSDK_AUTH_ACCESS_TOKEN

If none is set, it falls back to gcloud’s application_default_credentials.json, the block’s 'adc' detection source. If that is missing too, the step is skipped and every block that references it with googleAuthId is blocked.

It also reads these variables, none of which counts as a credential:

• Project: CLOUDSDK_CORE_PROJECT, GOOGLE_CLOUD_PROJECT, GOOGLE_PROJECT, GCLOUD_PROJECT
• Region: CLOUDSDK_COMPUTE_REGION, GOOGLE_CLOUD_REGION
• Zone: CLOUDSDK_COMPUTE_ZONE

When none of those is set, the block’s own project, defaultRegion, and defaultZone props are used. The app works the other way around: the project prop wins over a detected project, and the region and zone come from the block’s region picker, which starts at defaultRegion, and from defaultZone, not from these variables.

On success the runner writes the project, region, and zone under every name that gcloud, the client libraries, and the OpenTofu google provider each look for, as the app does: GOOGLE_CLOUD_PROJECT / CLOUDSDK_CORE_PROJECT / GOOGLE_PROJECT, GOOGLE_CLOUD_REGION / CLOUDSDK_COMPUTE_REGION / GOOGLE_REGION, and CLOUDSDK_COMPUTE_ZONE / GOOGLE_ZONE. A credentials file also sets CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE, so gcloud uses that file and ignores any account from gcloud auth login. An access token is written as both GOOGLE_OAUTH_ACCESS_TOKEN and CLOUDSDK_AUTH_ACCESS_TOKEN, and clears the override.

Two things differ from the session environment the app writes:

• Inline JSON in GOOGLE_CREDENTIALS is passed on as it is, and GOOGLE_APPLICATION_CREDENTIALS and CLOUDSDK_AUTH_CREDENTIAL_FILE_OVERRIDE are cleared. So only tools that read GOOGLE_CREDENTIALS, such as the OpenTofu google provider, find the credential, and gcloud and the client libraries don’t. The app writes the JSON to a file and points both variables at it.
• CLOUDSDK_CORE_ACCOUNT isn’t set.

Using a test prefix

As with AwsAuth, set env_prefix on the test step to keep test credentials apart from your local ones:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: google-auth
        env_prefix: RUNBOOKS_TEST_
        expect: success

The runner then checks:

• RUNBOOKS_TEST_GOOGLE_APPLICATION_CREDENTIALS
• RUNBOOKS_TEST_GOOGLE_CREDENTIALS
• RUNBOOKS_TEST_GOOGLE_OAUTH_ACCESS_TOKEN
• RUNBOOKS_TEST_CLOUDSDK_AUTH_ACCESS_TOKEN
• RUNBOOKS_TEST_CLOUDSDK_CORE_PROJECT (and the other project/region/zone names above, prefixed)

If no prefixed credential is found, it falls back to the unprefixed variables, and then to gcloud’s ADC file. A prefixed source in the app never falls back, so a test whose prefixed variables are missing can still pass on your own Google credentials.

CI example

.github/workflows/test.yml

- name: Authenticate to Google Cloud
  id: auth
  uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: projects/123/locations/global/workloadIdentityPools/ci/providers/github
    service_account: runbooks-ci@my-project.iam.gserviceaccount.com

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_TEST_GOOGLE_APPLICATION_CREDENTIALS: ${{ steps.auth.outputs.credentials_file_path }}
    RUNBOOKS_TEST_CLOUDSDK_CORE_PROJECT: my-project
    RUNBOOKS_TEST_CLOUDSDK_COMPUTE_REGION: us-central1

Testing GitHubAuth blocks

For a <GitHubAuth> block, the test runner looks for a token for the block’s host: the one its host prop names, or github.com without one. It ties each environment variable to a host the way the app does, so a token is only used for its own host:

• For github.com: RUNBOOKS_GITHUB_TOKEN (recommended for tests), then GITHUB_TOKEN, then GH_TOKEN. The last two count only while GH_HOST is unset or names github.com or a GitHub Enterprise Server host.
• For a *.ghe.com tenant: GITHUB_TOKEN, then GH_TOKEN, when GH_HOST names the tenant.
• For a GitHub Enterprise Server host: GH_ENTERPRISE_TOKEN, then GITHUB_ENTERPRISE_TOKEN, when GH_HOST names the host.

On success, the blocks after it see GITHUB_TOKEN and GITHUB_HOST. The app also sets GITHUB_USER and GH_HOST, and for an Enterprise Server host GH_ENTERPRISE_TOKEN, so under test a script has those only if the environment the tests run in sets them.

A host that isn’t a valid host is a configuration error in the app. Under test, the block uses github.com instead.

Using a test prefix

With env_prefix on the step, the runner reads the prefixed names:

runbook_test.yml

tests:
  - name: happy-path
    steps:
      - block: gh-auth
        env_prefix: CI_
        expect: success

This step reads CI_GITHUB_TOKEN and CI_GH_TOKEN, or for an Enterprise Server host CI_GH_ENTERPRISE_TOKEN and CI_GITHUB_ENTERPRISE_TOKEN. CI_GH_HOST ties them to a host, the way GH_HOST does for the unprefixed names. RUNBOOKS_GITHUB_TOKEN isn’t read with a prefix.

CI example

.github/workflows/test.yml

- name: Run runbook tests
  run: runbooks-cli test ./runbooks/...
  env:
    RUNBOOKS_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Testing GitLabAuth and GitAuth blocks

For a <GitLabAuth> block, the test runner looks for a GitLab token in these environment variables, in order:

1. GITLAB_TOKEN
2. GITLAB_ACCESS_TOKEN
3. OAUTH_TOKEN

With env_prefix on the step, it checks the prefixed names instead (CI_GITLAB_TOKEN, CI_GITLAB_ACCESS_TOKEN, CI_OAUTH_TOKEN). The app reads only the first two of those with a prefix.

On success, the blocks after it see GITLAB_TOKEN and GITLAB_HOST. As in the app, a token from the environment belongs to one GitLab host: the one GITLAB_HOST (or GITLAB_URI, GL_HOST) names, or gitlab.com if none is set. A block that pins another host with instanceUrl or host skips unless GITLAB_HOST names that host too, and a block whose pinned value isn’t a valid URL or host always skips. Unlike the app, the runner ties a prefixed token to a host with these unprefixed variables too. The app uses the prefixed ones, such as CI_GITLAB_HOST.

The test runner never sends the token anywhere, so a fake one is enough to test that the block authenticates:

runbook_test.yml

tests:
  - name: detects-env-token
    env:
      GITLAB_TOKEN: fake-token
    steps:
      - block: gitlab-auth
        expect: success

A <GitAuth> block is tested like <GitHubAuth> or <GitLabAuth>, depending on its provider prop (github by default).

Testing GitClone blocks

A <GitClone> clones its prefilledUrl into the working directory. With source="local" or a prefilledRepoDir, it uses that checkout instead. A block without the prop it needs has nothing to clone, so its step is skipped. A test with no steps expects every block to succeed, so give the block its prefilledUrl or prefilledRepoDir, or list it as a step with expect: skip.

The block’s outputs under test differ from the app’s:

• It sets clone_path and file_count, and ref for a clone with a prefilledRef. file_count and ref exist only under test.
• It doesn’t set repo_owner, repo_name, org_id or repo_id. A block that references one of them fails under test, even behind a hasKey guard.

Each <GitClone> that succeeds also becomes the repository that $REPO_FILES and target="worktree" use, so with several of them the last one to run wins. In the app, the first repository added, whether by a clone or a local checkout, stays in use until the user selects another.

Cloning with a token

A <GitClone> that references an auth block with gitAuthId or githubAuthId clones an https URL with that block’s token when the URL is on the host the block authenticated to. That includes GitHub Enterprise Server and self-managed GitLab hosts. The username is oauth2 for GitLab and x-access-token for GitHub.

Without a reference, the runner recognizes github.com, *.ghe.com, and gitlab.com URLs and uses that provider’s token from the test case environment: GITHUB_TOKEN or GH_TOKEN for GitHub, GITLAB_TOKEN for GitLab. A token already set where the tests run counts too, so to clone without one, blank it in the test case’s env:

tests:
  - name: anonymous-clone
    env:
      GITHUB_TOKEN: ""
      GH_TOKEN: ""

Testing pull request blocks

<GitPullRequest>, <GitHubPullRequest>, and <GitLabMergeRequest> blocks never run in tests, since they would push a branch and open a real pull request. A step that names one must use expect: skip, or expect: blocked when the auth block it references hasn’t run or was skipped. A test with no steps expects them to skip.

runbook_test.yml

tests:
  - name: skip-without-credentials
    steps:
      - block: gh-auth
        expect: skip
      - block: create-pr
        expect: skip

Testing Finish blocks

A <Finish> with a final check (command or path) runs it like a <Check>: exit code 0 is success, 2 is warn, and anything else is fail. A <Finish> without a final check has nothing to run, so its step succeeds straight away. Either way, the test can’t see the celebration.

runbook_test.yml

tests:
  - name: final-check-fails-without-setup
    steps:
      - block: setup
        expect: skip
      - block: finish
        expect: fail

Testing Prerequisites blocks

A step for a <Prerequisites> block looks for its tools and reads their versions as the app does, on the PATH the test runs with: the environment runbooks test started with, the test’s env, and what earlier steps exported. The step’s status is the check’s: success when every tool is found (one tool of an anyOf list is enough), warn when only optional ones aren’t, and fail when a required one isn’t. A step that succeeds or warns publishes the block’s outputs.

Unlike the app, the test doesn’t run your login shell to read PATH again when a tool is missing. To test a check that a runbook’s install step makes pass, run the block before and after that step:

runbook_test.yml

tests:
  - name: install-then-check-again
    steps:
      - block: tools
        expect: fail
      - block: install-tool
        expect: success
      - block: tools
        expect: success

Testing Iframe blocks

Tests can’t run the page an <Iframe> block shows, so a step for the block stands in for the page: its set_outputs become the block’s outputs, as if the page had set them. Only an <Iframe> with an id can be a step.

runbook_test.yml

tests:
  - name: deploy-to-picked-region
    steps:
      - block: region-picker
        set_outputs:
          region: eu-west-1
      - block: deploy
        expect: success

A step whose set_outputs names an output the block’s outputs prop doesn’t list fails, just as the app refuses that output from the page. Without set_outputs, the step succeeds and sets nothing, so blocks that read the block’s outputs stay blocked. A test with no steps runs each <Iframe> with an id that way.

Testing DirPicker blocks

The runner doesn’t run <DirPicker> blocks. A step that names one fails the test with references unknown block, and a test with no steps leaves the block out. Its PATH output never exists under test, so a block that reads it can only be tested with expect: blocked or expect: skip.

Running tests

The runner executes each test case’s blocks in a temporary directory by default, captures script output and generated files, and checks them against the test’s assertions. For a failed test it prints which block failed and why.

If runbook.mdx isn’t valid MDX, or contains something that Runbooks refuses to render, the runner fails it before any block runs and names the line with the error.

The runner is the same version as the app it comes with, which runbooks-cli --version prints. Like the app, it fails a runbook whose front matter requires another version of Runbooks before any block runs, and fails every test of a runbook with a block this version doesn’t have.

It exits with status 0 if every test passed. It exits with 1 if any test failed, a runbook_test.yml couldn’t be loaded, or no runbook has the test case named with --test.

Single runbook

# Run all tests for a runbook
runbooks-cli test ./my-runbook

# Run a specific test case
runbooks-cli test ./my-runbook --test happy-path

# Verbose output
runbooks-cli test ./my-runbook -v

Multiple runbooks

End the path with /... to test every runbook under a directory:

# Test all runbooks in a directory tree
runbooks-cli test ./runbooks/...

Runbooks are tested one at a time. The --max-parallel flag is accepted but currently has no effect.

Warnings

The runner warns about each <Command>, <Check> and <Finish> script that inserts {{ .inputs.X }} or {{ .outputs.B.K }} into its source, since env passes those values as data. The warnings appear under the runbook’s name and are counted in the summary line. In a JUnit report they go in the suite’s <system-err>. A warning doesn’t fail a test.

CI output

To write a JUnit XML report:

runbooks-cli test ./runbooks/... --output junit --output-file results.xml

The file is written before the command exits, including when tests fail. If it can’t be written, the CLI prints the error and writes the report to stdout instead.

Out-of-order testing

To run blocks in a different order than the runbook defines them, list them under steps. Steps run in the order listed, and a block listed more than once runs once per step, each time with that step’s expectation.

A block added to the runbook later is not tested until you add it to steps. Leave steps empty where you can, so new blocks are covered without a change to the test.

tests:
  - name: out-of-order
    description: Test dependency enforcement

    steps:
      # This should be blocked because create-account hasn't run
      - block: create-resources
        expect: blocked
        missing_outputs:
          - outputs.create_account.account_id

      # Run the dependency
      - block: create-account
        expect: success

      # Now this should succeed
      - block: create-resources
        expect: success

Example workflow

These jobs run in your own repository, which has no copy of the test runner. So each one checks out the Runbooks repository into .runbooks, builds the runner there, and then runs it against the runbooks in your repository’s runbooks directory. Point the runner at that directory rather than ./..., which would also find the test runbooks in .runbooks.

GitHub Actions

.github/workflows/test-runbooks.yml

name: Test Runbooks

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read

    steps:
      - uses: actions/checkout@v4

      # The test runner is built from the Runbooks repository.
      # Set `ref` to a release tag to pin the version.
      - uses: actions/checkout@v4
        with:
          repository: gruntwork-io/runbooks
          path: .runbooks

      - uses: jdx/mise-action@v2
        with:
          working_directory: .runbooks

      - name: Build the test runner
        working-directory: .runbooks
        run: |
          bun install --frozen-lockfile
          just compile-test-cli

      - name: Configure AWS credentials (OIDC)
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ vars.AWS_ROLE_ARN }}
          aws-region: ${{ vars.AWS_REGION }}

      - name: Run runbook tests
        run: .runbooks/resources/bin/runbooks-test test ./runbooks/... --output junit --output-file results.xml
        env:
          RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: ${{ env.AWS_ACCESS_KEY_ID }}
          RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: ${{ env.AWS_SECRET_ACCESS_KEY }}
          RUNBOOKS_TEST_AWS_SESSION_TOKEN: ${{ env.AWS_SESSION_TOKEN }}
          RUNBOOKS_TEST_AWS_REGION: ${{ vars.AWS_REGION }}

      - name: Upload test results
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: results.xml

GitLab CI

.gitlab-ci.yml

test-runbooks:
  stage: test
  variables:
    RUNBOOKS_TEST_AWS_ACCESS_KEY_ID: $AWS_ACCESS_KEY_ID
    RUNBOOKS_TEST_AWS_SECRET_ACCESS_KEY: $AWS_SECRET_ACCESS_KEY
    RUNBOOKS_TEST_AWS_REGION: $AWS_REGION
  before_script:
    # The test runner is built from the Runbooks repository.
    # Add `--branch <release tag>` to pin the version.
    - git clone --depth 1 https://github.com/gruntwork-io/runbooks.git .runbooks
    - curl -fsSL https://mise.jdx.dev/install.sh | sh
    - export PATH="$HOME/.local/bin:$HOME/.local/share/mise/shims:$PATH"
    - cd .runbooks
    - mise install
    - bun install --frozen-lockfile
    - just compile-test-cli
    - cd "$CI_PROJECT_DIR"
  script:
    - .runbooks/resources/bin/runbooks-test test ./runbooks/... --output junit --output-file results.xml
  artifacts:
    reports:
      junit: results.xml

Testing with external dependencies

For runbooks that call external services such as GitHub or AWS, test in tiers. Each tier covers more than the one before it and is slower and harder to set up, as in the test pyramid.

Tier 1: template validation

Tier 1 checks that templates generate files and makes no external calls.

tests:
  - name: template-validation
    description: Validate template generation

    inputs:
      config.ProjectName: "test-project"
      config.Region: "us-east-1"

    steps:
      - block: generate-config  # Template block
        expect: success
      - block: deploy-resources  # Skip external calls
        expect: skip

    assertions:
      - type: files_generated
        block: generate-config
        min_count: 1

Tier 2: dry-run mode

Tier 2 runs scripts with their external calls switched off. The script checks an environment variable, and the test sets it. RUNBOOK_DRY_RUN is a convention, and any variable name works.

In the Command or Check script:

#!/bin/bash
set -e

# Dry-run support
DRY_RUN="${RUNBOOK_DRY_RUN:-false}"

if [[ "$DRY_RUN" == "true" ]]; then
    echo "[DRY-RUN] Would create PR in $GITHUB_ORG/$REPO_NAME"
    echo "[DRY-RUN] gh pr create --title '$TITLE'"
    exit 0
fi

# Real execution
gh pr create --title "$TITLE" --body "$BODY"

In the test configuration:

tests:
  - name: dry-run-flow
    description: Test full flow without real API calls

    env:
      RUNBOOK_DRY_RUN: "true"

    steps:
      - block: generate-config
        expect: success
      - block: create-pr
        expect: success  # Succeeds in dry-run mode

Tier 3: integration tests

Tier 3 runs against real services. Set credentials as environment variables in CI, and run these tests manually or on a schedule:

tests:
  - name: integration-test
    description: Full integration (requires credentials)

    env:
      # Credentials set in the CI environment are available to the test,
      # so they don't need to be set here.

    steps:
      - block: deploy-resources
        expect: success

    cleanup:
      - path: scripts/cleanup-test-resources.sh

Best practices

• Start with one test case and no steps, so every block runs in document order. Add inputs, steps, and assertions from there.
• Cover the failure paths too: expect: fail for a script that should exit non-zero, expect: blocked with missing_outputs for a block whose dependency hasn’t run, and expect: config_error with error_contains for a misconfigured block.