Skip to content

Add reboot.bdd: write Reboot tests in Gherkin with pytest-bdd - #149

Draft
benh wants to merge 5 commits into
mainfrom
pytest-bdd
Draft

Add reboot.bdd: write Reboot tests in Gherkin with pytest-bdd#149
benh wants to merge 5 commits into
mainfrom
pytest-bdd

Conversation

@benh

@benh benh commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

Adds reboot.bdd, a pytest-bdd layer for testing Reboot applications with Gherkin, plus its test suites.

What a test looks like

Feature: Accounts

  Background:
    Given the application is up

  Scenario: Depositing adds to the balance
    Given an `Account` for "alice" gets created via `open` with `initial_balance: 100`
    When the `Account` for "alice" gets a `deposit` with `amount: 50`
    Then the result has `updated_balance: 150`
    And `balance` on the `Account` for "alice" has `balance: 150`

The test module does from reboot.bdd.steps import * and defines an application fixture returning the Application under test. Each scenario runs against a fresh started Reboot harness, and each step's call runs on a fresh ExternalContext (the way each external call in production arrives with its own) unless the scenario creates one to share via Given a shared context.

Built-in steps

  • Given the application is up
  • Given a shared context
  • Given an `Account` for "alice" gets created via `open` with `initial_balance: 100` (usable as When too)
  • When the `Account` for "alice" gets a `deposit` with `amount: 50` (usable as Given too)
  • When the `Account` for "bob" attempts a `withdraw` with `amount: 50`
  • Then the attempt aborts with `OverdraftError` where `amount: 20`
  • Then `balance` on the `Account` for "alice" has `balance: 150`
  • Then `balance` on the `Account` for "ghost" aborts with `StateNotConstructed`
  • When `get_owner` on the `Account` for "frank" has `owner.name` saved as "$owner_name" (later steps say $owner_name in a state's ID or as a property value)
  • Then the result has `updated_balance: 150` (as Given/When, saves instead)

The property grammar

A has/where/with list holds JSON object members, `name: value` each in backticks, separated by commas or and. Values are JSON (an object or array calls through the method's request type; on assertions it compares as the complete message); a dotted name nests when calling and reaches into the response when asserting; a malformed or unknown property raises instead of being skipped. In a has or where list a clause can instead be `name` saved as "$name": a Then has/where asserts and refuses saving clauses, a Given/When has saves and refuses comparing clauses.

Readers are only read via `reader` on ...; gets a/attempts a refuse them with a pointing error, and the reader steps refuse writers likewise (whether a method is a reader comes from the generated reactively() surface, which serves exactly the unary readers). Reading records the result, so "the result" always means the most recent call any step made.

State types are resolved from the Application's servicers by class name, or by full state type name (e.g. bank.v1.Account) when more than one state type goes by the class name; both proto and pydantic codegen are supported.

How it works

pytest and pytest-bdd are synchronous while everything Reboot is async, and we deliberately don't use pytest-asyncio (pytest-bdd never awaits step functions, so it wouldn't help anyway). Instead, each scenario runs one event loop on a background thread, created before its first step and closed after its last: one loop per scenario, matching both unittest.IsolatedAsyncioTestCase (one loop per test) and production (one application runs on one event loop under rbt dev run and rbt serve), so nothing a scenario leaks can run on into later scenarios. The loop's teardown mirrors IsolatedAsyncioTestCase: cancel pending tasks, shut down async generators, close.

The reboot.bdd given/when/then/step decorators work like pytest-bdd's except that the decorated step function may be async def, so custom developer steps get the same treatment as the built-ins (exercised by the custom steps in tests/reboot/bdd/bdd_tests.py and tests/reboot/bdd/pydantic/bdd_tests.py).

Also in this PR

  • Reboot.stop() now cancels (via wait_for_tasks) the monitor_event_loop() task that start() creates. IsolatedAsyncioTestCase hid this leak by closing its per-test loop; on a longer-lived loop every harness left a pending task behind. Ran //tests/reboot:external_context_tests_py to check for regressions, but this touches every harness user, so please look closely.
  • pytest==8.4.2 and pytest-bdd==8.1.0 added to reboot/requirements.in (and mypy.ini ignore sections for both).

Testing

bazel test //tests/reboot/bdd:bdd_tests_py //tests/reboot/bdd/pydantic:bdd_tests_py (proto and pydantic suites, plus unit tests for name collisions, unknown properties, and reader detection).

Not yet done (follow-ups)

  • The published wheel should probably get pytest-bdd as a reboot[bdd] extra rather than a hard dependency; only the Bazel side is wired here.
  • Docs (a testing-bdd.md skill reference), state assertions via a test-only raw-state read (e.g. Then the state of the `Account` for "alice" has `balance: 70` ), reactive eventually variants of the assertion steps, identity/auth steps, task steps, and failure/recovery steps.

🤖 Generated with Claude Code

https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m

@aviator-app

aviator-app Bot commented Sep 1, 2026

Copy link
Copy Markdown

Current Aviator status

Aviator will automatically update this comment as the status of the PR changes.
Comment /aviator refresh to force Aviator to re-examine your PR (or learn about other /aviator commands).

This pull request is currently open (not queued).

How to merge

To merge this PR, comment /aviator merge or add the mergequeue-ready label.


See the real-time status of this PR on the Aviator webapp.
Use the Aviator Chrome Extension to see the status of your PR within GitHub.

`start()` creates a `monitor_event_loop()` task but `stop()` never
cancelled it. Tests based on `IsolatedAsyncioTestCase` hide the leak
because each test's event loop closes right after `stop()`, but on a
long-lived event loop (as `reboot.bdd` uses) every harness left a
pending task behind that warned at garbage collection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m
@benh
benh force-pushed the pytest-bdd branch 25 times, most recently from 9aa02e5 to 28ca315 Compare September 2, 2026 03:29
benh and others added 2 commits September 2, 2026 03:34
Developers write Gherkin scenarios against built-in steps, e.g.:

    Given the application is up
    And an `Account` for "alice" gets created via `open` with
      `initial_balance=100`
    When the `Account` for "alice" gets a `deposit` with `amount=50`
    Then `balance` on the `Account` for "alice" has
      `balance=150`

A test module brings in the built-in steps and the fixtures they run
on with `from reboot.bdd.steps import *` and defines an `application`
fixture returning the `Application` under test. Each scenario runs
against a fresh started `Reboot` harness on its own event loop, one
loop per scenario the way one application runs on one event loop
under `rbt dev run` and `rbt serve`. Each step's call runs on a fresh
`ExternalContext`, the way each external call in production arrives
with its own, unless the scenario creates one to share via
'Given a shared context'.

Custom steps may be `async def`: the `reboot.bdd`
`given`/`when`/`then` decorators run them on the scenario's event
loop, the same loop the harness and the built-in steps run on, which
is what lets `reboot.bdd` work under plain pytest without
`pytest-asyncio`.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m
`reboot.bdd` resolves state types and calls methods the same way for
proto and pydantic codegen, since both come from the same template;
this pins the pydantic path with its own `Account` mirroring
`tests/reboot/bdd/accounts.feature`, plus a custom `async def` step
that calls through `World.call()` instead of the generated code.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m
@benh
benh force-pushed the pytest-bdd branch 11 times, most recently from ce87dc2 to 22bfb0b Compare September 2, 2026 19:32
Six of the example test suites take a value out of one response, e.g.
an account or order ID, and use it in the next call. In Gherkin, a
'has' or 'where' list is a list of clauses: a comparing clause
asserts, and a saving clause saves a property under a name:

    When `get_owner` on the `Account` for "frank" has
      `owner.name` saved as "$owner_name"

A Then 'has' or 'where' asserts and refuses saving clauses; a Given
or When 'has' saves and refuses comparing clauses. Later steps say
`$name` to use a saved value, in a state's ID or as a property
value; a quoted "$name" stays the literal string, and saved values
are used as-is, so a saved message can be passed straight into a
later call's properties.

Readers are only read via '`reader` on ...', and a reader's abort
gets its own assertion, 'aborts with `SomeError` where ...';
'gets a' and 'attempts a' refuse readers, pointing at those steps,
the way they refuse writers. Whether a method is a reader comes from
the generated `reactively()` surface, which serves exactly the unary
readers. Reading records the result, so 'the result' always
means the most recent call any step made.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m
@benh
benh force-pushed the pytest-bdd branch 10 times, most recently from d560abf to a0f3238 Compare September 3, 2026 03:42
Properties now read as the members of one JSON object, `name: value`
each in backticks, and an object or array value goes through the
named method's request type, looked up via the generated client
class's `<Method>Request` alias: pydantic requests validate with
`model_validate` and proto requests parse with
`json_format.ParseDict`, so nested messages, enums by name, and
64-bit integers all follow the JSON semantics those types define:

    When the `Account` for "frank" gets a `set_owner` with
      `owner.name: "Frankie"` and `owner.tags: ["pro"]`

A dotted name nests when calling, and a name that collides with
another, e.g. both `owner` and `owner.name`, raises. On assertions,
an object compares as the complete message the actual value's type
parses it as, and an array compares elementwise; a saved property
whose value is already a message is merged into the request as-is.
Also guards pydantic request validation against its default of
ignoring unknown keys, which made a mistyped property a silent
no-op.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015whh5EmdU8PAoxSevMRn5m
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant