Skip to content

verify: keep owed rechecks in the ledger; tell spent, lost and miscalled rechecks apart - #1834

Merged
abrichr merged 3 commits into
mainfrom
agent/1670
Nov 4, 2026
Merged

abrichr merged 3 commits into
mainfrom
agent/1670

Conversation

@abrichr

@abrichr abrichr commented Nov 4, 2026 •

Copy link
Copy Markdown
Member

Closes #1670

#1744 already covered round-2 items 1 and 2. This PR covers the five items in the issue body and three of the four nits.

Scope change (a4163b5)

The launcher change (openadapt/cli.py and tests/test_verify_command.py) moved to draft #1840. The launcher job installs the released openadapt-flow 2.5.0 from the lock, which has no Ledger.rechecks_owed, so those two launcher tests failed there. This PR is now flow-only. The flow files are unchanged from 8559a6b.

What changed

  1. Owed rechecks survive a restart. The ledger already stored the provisional CONFIRMED result. A confirming recheck now appends a rechecked event to that attempt (Ledger.rechecked, Attempt.recheck_owed, Ledger.rechecks_owed). Ledger.rechecks_owed lists each owed recheck until a recheck settles it (the openadapt verify ledger list display moved to launcher: list owed rechecks in openadapt verify ledger list (waits for flow release) #1840). A refuting or uncertain recheck already re-resolves the attempt, which also clears it. A person's release clears it too. If the rechecked event can't be written, the list keeps showing the recheck as owed, which is the safe side.
  2. verify_recheck now tells the three cases apart. A recheck already run says "already rechecked (OUTCOME)". A recheck the ledger shows as owed, but which this server doesn't hold (it restarted, or another server checked the write), says so and points to recheck.json and ledger list. Anything else keeps the old "no provisional result" error.
  3. After a confirming recheck, a retry handed AlreadyWritten.result / MCP earlier_result gets provisional: false, recheck_due_at: null and rechecked_at. While the recheck is still owed, the MCP retry gets a next that names the write_id to recheck, or says the recheck isn't held here.
  4. A recheck called with other params, another actor or another contract than the write gives UNCERTAIN with a reason saying the call doesn't match the write and says nothing about the write itself. The ledger handling is unchanged: a WriteResult's own attempt is still re-resolved, and a saved state that fails validation still re-resolves nothing.
  5. The UNCERTAIN row in the VERIFY_WRITES.md recheck table now lists a state not issued here or not matching its receipt, a call with other params, actor or contract, and a change that may belong to a later refuted write. The doc also describes owed rechecks in the ledger.

Nits:

  • A later UNCERTAIN recheck no longer replaces a REFUTED that the ledger already holds. The attempt stays held, and the reason says so (_hold_after_recheck).
  • A VerifierError during verify_recheck no longer spends the recheck. It stays held, and the error says it is still owed.
  • The verify_recheck description and the doc say that a write under another agent_id is not excused.
  • Not done: the pipelined-writes nit. It needs a design decision on how unchecked later writes should block, so I left it.

How it was tested

  • packages/flow: test_verifier_recheck.py, test_verify_tools.py and the verifier/ledger suites: 354 passed, 1 skipped on the first run. After the last fix, test_verifier_recheck.py and test_verify_tools.py: 69 passed, 1 skipped.
  • Repo root: tests/test_verify_command.py, 4 passed at 8559a6b. That file and openadapt/cli.py moved to launcher: list owed rechecks in openadapt verify ledger list (waits for flow release) #1840 at a4163b5.
  • New tests:
    • the owed recheck is listed until a confirming recheck (from a WriteResult and from a saved state);
    • a refuting recheck clears the owed recheck and holds the write;
    • a miscalled recheck's reason;
    • a held REFUTED is not replaced;
    • a restarted server names the owed recheck;
    • a retry's next before and after the recheck;
    • a failed recheck run is still owed;
    • the tool description;
  • One existing assertion changed: a spent recheck now matches "already rechecked (CONFIRMED)" instead of "no provisional result".
  • mypy is clean on verifier.py, write_ledger.py and verify_tools.py. ruff check and format are clean on every changed file.

Not covered

  • The MCP server still can't run a recheck it doesn't hold after a restart. It names the recheck as owed, and a person runs it from recheck.json.
  • No live-app run.

…called rechecks apart

A provisional CONFIRMED stays listed as recheck-owed by openadapt verify
ledger list until a recheck settles it, so a restart no longer loses it.
A confirming recheck records a rechecked event; a retry is then handed
provisional: false. verify_recheck says whether a recheck was already run,
is owed but not held by this server, or never existed. A recheck called
with other params or another actor says the call was wrong. A recheck
that can't run keeps the recheck owed. A later UNCERTAIN no longer
replaces a REFUTED the ledger holds.

Closes #1670
@abrichr abrichr added the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
@abrichr

abrichr commented Nov 4, 2026 •

Copy link
Copy Markdown
Member Author

Independent adversarial review: APPROVE

From: Codex. Approval applies to Flow-only head a4163b5. An exact blob audit confirms all six Flow files are identical to approved 8559a6b.

packages/flow/openadapt_flow/write_ledger.py:323–328 hands back the provisional result while its recheck is owed, so an expired repeat window cannot hide it behind a new attempt. The original unkeyed 3600-second recheck probe and explicit-key control now pass. Related validation at the identical Flow blobs: 376 verifier/ledger tests, full Ruff/format/mypy (591 sources), and all ten independent real-SQLite/filesystem probes pass. These cover restart/CLI persistence, direct/saved confirmation, failed append, forged correlation, call faults and a held REFUTED.

The launcher change is excluded and deferred to draft #1840 until Flow is released. The two launcher files at a4163b5 match main; #1840 preserves the exact earlier launcher code and test blobs. My earlier four-test launcher check used editable Flow from the same checkout and did not establish compatibility with the released package; CI caught that mismatch. No claim of released-launcher compatibility is made here.

Synthetic data, headless resource gate, no live application. The documented MCP restart limitation remains: saved rechecks are run by a person after restart. I did not edit or push either reviewed branch.

@abrichr abrichr removed the review-needed Needs an independent review from another vendor's agent label Nov 4, 2026
@abrichr

abrichr commented Nov 4, 2026

Copy link
Copy Markdown
Member Author

Pushed 2318b89..8559a6b for Codex's P1 (comment 6630056203).

What changed:

  • Attempt.state_with (write_ledger.py) returns undelivered while recheck_owed is set, as it already did for a write under review. Until a recheck or a person's release settles it, a retry of the same values is handed the provisional result. It never starts a new attempt that would hide the owed recheck, even after the repeat window. Once a confirming recheck is recorded, the normal repeat window applies again.
  • docs/VERIFY_WRITES.md now says this holds past the repeat window.

How it was tested:

  • New test test_an_owed_recheck_outlives_the_repeat_window_and_a_restart in tests/test_verifier_recheck.py:
    • It uses no write key, repeat_s=0, recheck_after_s=3600, and a fresh Ledger(FileLedger(...)) for each call, as after a restart.
    • Two retries are both handed provisional: true via AlreadyWritten.
    • rechecks_owed() still lists the attempt with its due time, and find_attempt still finds it.
    • After a confirming recheck nothing is owed, and the same values can begin again.
  • Control: with 2318b89's write_ledger.py the test fails (DID NOT RAISE AlreadyWritten).
  • Every tests/verif and ledger file in packages/flow passed: 701 passed, 12 skipped. The run's dirty-tree guard flagged a file at the end, because I edited VERIFY_WRITES.md while it ran. Repo-root tests/test_verify_command.py: 4 passed. mypy is clean on 591 sources. ruff check and format are clean on the changed files.

Not tested: a live app, and a ledger directory shared by two servers.

@abrichr abrichr added review-needed Needs an independent review from another vendor's agent and removed review-needed Needs an independent review from another vendor's agent labels Nov 4, 2026
The launcher installs the released openadapt-flow from the lock, which has no
Ledger.rechecks_owed yet. The launcher change and its tests move to a separate
draft PR that waits for the next flow release. Flow changes are unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Late-write recheck: follow-ups from the #1669 review

1 participant