Commit message test plans

• Software engineers who already write test plans in commit messages or code review descriptions.

• People working with patch stacks or stacked diffs.

Intended audience
Origin	Private correspondence re Julio Merino's post A markdown-based test suite.
Mood	Practical.

• Why

• How

  • Patterns

  • Script

Julio Merino recently published A markdown-based test suite, about using Markdown itself as a lightweight test format.

That reminded me of a related workflow I’ve been using for a while with scrut: I put executable test plans directly in commit messages.

Why

• Makes the commit message’s test plan executable instead of purely descriptive.

  • Great for test-driven development, to ensure that my validation plan actually detects the underlying issue.

  • Great for knowledge sharing and onboarding teammates.

• Supports re-validating entire commit stacks via git test.

  • Often useful when rebasing on top of upstream changes.

  • On failure, it makes it quick and easy to bisect the first broken commit.

• Supports ad-hoc and differential testing, where there is no tested correct output, and we just want to document changes.

How

Inside my commit messages, I add scrut code blocks with test commands to run. Example:

fix(tests): fix tests on macOS with Git v2.37

...

Test Plan
---------

```scrut
$ cargo nextest run --workspace --no-fail-fast -- 'submodule'

I use a small script called `git-test-message` to read the commit message and run the `scrut` tests in the repository working tree:

For individual runs, I invoke it like this:

$ git test-message 🔎 Found 1 test document(s)

Result: 1 document(s) with 1 testcase(s): 1 succeeded, 0 failed and 0 skipped

With [`git test`](https://github.com/arxanas/git-branchless/wiki/Command:-git-test), I’ve configured it as my default test command, which runs it on the entire stack:

$ git config 'branchless.test.alias.default' git test-message @

$ git test run ✓ Passed (cached): 624edd2 fix(tests): fix tests on macOS with Git v2.37 Ran command on 1 commit: git test-message @ 1 passed, 0 failed, 0 skipped

### Patterns

By default, `scrut` asserts that the command exits successfully and that `stdout` matches. For some tools, especially `bazel`, `stdout` is not interesting or is non-deterministic, so I often redirect it to `stderr` so that it’s not asserted, but is still logged on failure:

$ bazel test //foo >&2

For ad-hoc validation, when there’s no test case to cover a specific situation, I often pipe to `grep` or use `scrut`’s [output expectations](https://facebookincubator.github.io/scrut/docs/tutorial/output-expectations/):

$ bazel run //foo | grep bar some line with bar

For differential testing, I might record the new behavior, check out the previous commit, record the old behavior, and diff the two:

$ bazel run //foo >after && git checkout HEAD~ && bazel run //bar >before && diff before after ...diff output here... [1]

The `[1]` means that exit code `1` is expected from `diff`.

### Script

Here’s my `git-test-message` script:

#!/bin/bash set -euo pipefail

mise exec 'cargo:scrut' -- scrut test
--work-directory="${PWD}"
--match-markdown='*'
<(git show --no-patch --format='%B' "${1:-HEAD}")

Notes:

- My script uses [`mise`](https://mise.jdx.dev/) to just-in-time provision the `scrut` binary.

- By default, `scrut` works in a temporary directory. I oftentimes run commands that need th repo state, so I added `--work-directory=${PWD}`.

- By default, `scrut` only runs on Markdown input files. I specified `--match-markdown='*'` to match the [process substitution](https://tldp.org/LDP/abs/html/process-sub.html) filename (which usually ends up being a path like `/dev/fd/63`).

- `scrut` has features to auto-update the snapshot tests, but I haven’t integrated that (since they’d have to be written back to the Git commit message).

The following are hand-curated posts which you might find interesting.

Want to see more of my posts? Follow me on [Bluesky](https://bsky.app/profile/arxanas.bsky.social), [Mastodon](https://types.pl/@arxanas), or [Twitter](https://twitter.com/arxanas), or subscribe [via RSS](https://blog.waleedkhan.name/feed.xml).