Skip to content

fix(wallet): bound a reservation deadline by the observing clock, not only by the first push's - #528

Merged
MichaelTaylor3d merged 30 commits into
mainfrom
loop/525-clock-anchor-bound
Sep 3, 2026
Merged

fix(wallet): bound a reservation deadline by the observing clock, not only by the first push's#528
MichaelTaylor3d merged 30 commits into
mainfrom
loop/525-clock-anchor-bound

Conversation

@MichaelTaylor3d

@MichaelTaylor3d MichaelTaylor3d commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Closes #525.

The defect

reserve_pushed_bundle (sage/rpc.rs:2325) reads the clock ONCE and writes submitted_at = now and expires_at = now + RESERVATION_TTL_MS. One reading far in the future (F) stores a deadline decades out, and nothing in the system could retire it:

  • prune_reservations deletes on expires_at <= now, which never arrives.
  • fix(wallet): bound the total hold a repushed bundle can keep on its inputs #505's upsert clamp is MAX(stored.expires_at, MIN(excluded.expires_at, stored.submitted_at + C)), so a later push under the corrected clock R computes MAX(F+T, MIN(R+T, F+C)) = F+T -- unchanged.
  • coin_reservations.transaction_id REFERENCES pending_transactions ON DELETE CASCADE with foreign_keys(true), so the child row survives too, and reserved_coin_ids keeps excluding the coin from unreserved_unspent_coins for ever.
  • reset_chain_cache does not delete either table and refuses while the row exists.

The coin is frozen permanently with no recovery available inside the product.

The fix -- repair at OBSERVATION

reserve_spend's upsert is unchanged. A write-time-only fix was rejected on the record: nothing re-pushes an accepted bundle unprompted, so it would leave the headline no-re-push freeze permanent.

WalletDb::prune_reservations now runs two repair UPDATEs before its two existing DELETEs, all four in ONE write-first transaction (the write lock is taken before any read -- the same ordering reserve_client_coins documents as what prevents a SQLITE_BUSY mis-mapped to Unavailable).

  1. UPDATE pending_transactions SET submitted_at = now, expires_at = now + RESERVATION_TTL_MS WHERE expires_at > now + MAX_RESERVATION_HOLD_MS
  2. UPDATE client_coin_reservations SET expires_at_ms = now + CLIENT_RESERVATION_MAX_TTL_MS WHERE expires_at_ms > now + MAX_RESERVATION_HOLD_MS

Return value unchanged: rows_affected of the first DELETE only. A repaired row is not a retired bundle, and folding it in would inflate a count callers read as in-flight spends.

Why the predicate is a self-contradiction check, not a heuristic

INSERT writes expires_at = now + T; the DO UPDATE clamp writes at most submitted_at + C. So every healthy row satisfies expires_at <= submitted_at + C, and with submitted_at <= now it satisfies expires_at <= now + C. Hence expires_at > now + C implies submitted_at > now -- the row claims to have been submitted after the present moment.

The false-fire floor

Let E = expires_at - submitted_at = min(e + T, C) where e is the row's age at the last push. The detector fires iff now < submitted_at + E - C, so a backwards clock step of d at age e fires iff d > max(C - T, e).

The minimum backwards step that can trip the repair on a legitimately created row is C - T = 50 minutes, and more for an older row. A claim that a 1 ms backwards step trips an at-cap row is false: an at-cap row is ~50 minutes old, so it needs d > 50 min too.

The genuine cost, stated not argued away

Past that floor a live bundle's row IS re-anchored and gets only T more from the repair instant, where the unrepaired code would have held it until the clock caught up (d + E - e longer). So a backwards step exceeding MAX_RESERVATION_HOLD_MS - RESERVATION_TTL_MS can return a still-live bundle's inputs after one further TTL instead of on clock recovery. That is a real move toward the #348/#497 double-spend direction, bounded by a gross (>50 min) clock anomaly, and accepted because the alternative it replaces is a permanent, in-product-unrecoverable freeze -- the failure RESERVATION_TTL_MS' own doc and SPEC.md 18.9a both name as the worse one. This fix does not "never release anything early".

The four decisions inside the shape

  1. Grant now + T, not now + C -- precisely what an honest push at now would have written. now + C would hand a repaired row six times a healthy row's first hold.
  2. submitted_at is re-anchored, and this is not optional. Left at F, stored.submitted_at + C = F + C, so A repushed bundle re-arms its reservation from now, so a retrying caller can hold coins indefinitely #502's cap clause stops binding on that row for ever and a retrying caller renews the hold indefinitely -- resurrecting the A repushed bundle re-arms its reservation from now, so a retrying caller can hold coins indefinitely #502 lockout. This is not a regression of A repushed bundle re-arms its reservation from now, so a retrying caller can hold coins indefinitely #502's stable-anchor rule: that rule keeps the column out of DO UPDATE SET so a CALLER cannot move the anchor. This write is not a caller; it is the row's owner replacing a value the system has proven impossible with one from the currently trusted clock. Stated in the doc comment.
  3. Both detectors use MAX_RESERVATION_HOLD_MS, including the client one. Keying the client detector on its own 10-minute ceiling would let a ~5-minute backwards step re-clamp a healthy client hold and reopen the cross-process selection window dig_ecosystem#3127 closes. One hour is the longest hold of any kind this wallet grants. Pinned by a const _: () = assert!(CLIENT_RESERVATION_MAX_TTL_MS <= MAX_RESERVATION_HOLD_MS, ...) beside the constant, so the detector cannot silently become unsound if either moves.
  4. The client repair grants now + CLIENT_RESERVATION_MAX_TTL_MS, not now + T: there is no submitted_at column and the caller's requested TTL is unrecoverable, so it grants the ceiling a caller could legitimately have asked for. That errs long, the safe direction for a build-window hold. The asymmetry is stated in the comment.

Blast radius checked

prune_reservations callers, via grep (gitnexus indexes are stale by construction here -- commitsBehind on dig-node is in the hundreds and points at the primary checkout, so impact would return a false-safe zero):

  • rpc.rs:875, 893, 2961, 3228, 3695 -- all already prune before a reservation-sensitive read, and all are unaffected: the signature, the return value's meaning, and the two DELETE predicates are unchanged.
  • rpc.rs:1198 (reset_coin_db) -- the one entry point that did NOT prune, now does. Without it a user whose first action after correcting the clock is a reset gets ResetRefusal::SpendInFlight telling them to wait, and waiting alone triggers no prune. SpendInFlight's counting predicate (db.rs:3005) and its message text are deliberately unchanged: excluding contradicted rows from the count would let a reset wipe coins beneath a bundle that may genuinely be in a mempool, and the sentence "Wait for them to confirm or expire, then retry" is false today but becomes TRUE under this fix.
  • Test-only callers in db.rs.

Risk: MEDIUM. It is custody-adjacent (it decides when a coin returns to selection) but it strictly shrinks an unbounded hold to a bounded one, and its one early-release path is the >50-minute-clock-anomaly case stated above.

Evidence

RED, before the fix (cargo test -p dig-wallet --lib, real counts, not a zero-match run):

test sage::db::tests::the_clock_contradiction_repair_re_anchors_submitted_at_so_the_cap_still_binds ... FAILED
test sage::db::tests::the_clock_contradiction_repair_grants_one_fresh_ttl_not_the_maximum_hold ... FAILED
test sage::db::tests::the_clock_contradiction_detector_fires_strictly_past_the_maximum_hold ... FAILED
test result: FAILED. 0 passed; 3 failed; 0 ignored; 0 measured; 798 filtered out

test sage::db::tests::a_reservation_anchored_to_a_far_future_clock_stops_freezing_its_coin_for_ever ... FAILED
test sage::db::tests::a_client_hold_written_under_a_far_future_clock_is_repaired_and_then_lapses ... FAILED
test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 799 filtered out

The headline failure line: assertion left == right failed: one TTL after the repair the coin returns to selection; before this fix it never did.

GREEN, after: test result: ok. 800 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out (baseline at f0a9726 was 794 passed / 1 ignored; +6 new tests).

cargo fmt --check on the two touched files only: CLEAN. No workspace-wide cargo fmt --all.

Tests added (6)

test pins
a_reservation_anchored_to_a_far_future_clock_stops_freezing_its_coin_for_ever the headline freeze, asserted on unreserved_unspent_coins(None) -- the selection surface where the money is frozen, not a pending_transactions row count one layer below the decision
the_clock_contradiction_repair_grants_one_fresh_ttl_not_the_maximum_hold the AMOUNT of the grant, not just its existence
the_clock_contradiction_repair_re_anchors_submitted_at_so_the_cap_still_binds the re-anchor: a re-push at corrected + 5.5T must land on corrected + C; without the re-anchor it lands on corrected + 6.5T
a_client_hold_written_under_a_far_future_clock_is_repaired_and_then_lapses the client arm
a_one_millisecond_backwards_clock_step_repairs_nothing no-false-positive guard (a)
the_clock_contradiction_detector_fires_strictly_past_the_maximum_hold no-false-positive guard (b): both sides of the exact bound -- at now + C untouched, at now + C + 1 repaired

Mutation table (each run in full, then reverted)

mutation result named tests that went red
1. delete the UPDATE pending_transactions statement entirely 796 passed; 4 failed a_reservation_anchored_to_a_far_future_clock_stops_freezing_its_coin_for_ever, the_clock_contradiction_repair_grants_one_fresh_ttl_not_the_maximum_hold, the_clock_contradiction_repair_re_anchors_submitted_at_so_the_cap_still_binds, the_clock_contradiction_detector_fires_strictly_past_the_maximum_hold
2. grant now + MAX_RESERVATION_HOLD_MS instead of now + RESERVATION_TTL_MS 797 passed; 3 failed ..._grants_one_fresh_ttl_not_the_maximum_hold, a_reservation_anchored_to_a_far_future_clock_..., ..._detector_fires_strictly_past_the_maximum_hold
3. drop SET submitted_at = ?now 799 passed; 1 failed the_clock_contradiction_repair_re_anchors_submitted_at_so_the_cap_still_binds (uniquely)

Mutation 3 localises to exactly one test, which is the one it must. Mutations 1 and 2 overlap, as expected: mutation 1 removes the whole repair, so every test that observes any part of it goes red; mutation 2 is the strictly narrower "amount" mutation and is the one distinguished by ..._grants_one_fresh_ttl_....

Also in this PR

  • Corrected the existing revert-proof fixture a_repush_under_a_clock_that_stepped_backwards_never_shortens_a_live_hold. Its third push passed submitted_at = first while moving only expires_at backwards -- a state reserve_pushed_bundle cannot produce, since it writes both from one clock read. It is now the production-shaped submitted_at = first - 60_000, expires_at = first - 60_000 + T (well under the 50-minute detector floor, so the row must be untouched by the new repair). Its existing assertion row.expires_at == extended_at + RESERVATION_TTL_MS still holds unchanged, now for a reachable state.
  • SPEC.md 18.9a ("In-flight coin reservation", the reservation-contract section -- note the file re-uses the number 18.9a at two other points for the offer/DID-mint suite; that pre-existing collision was NOT touched or renumbered): a normative statement that a recorded deadline exceeding the current instant by more than MAX_RESERVATION_HOLD_MS MUST be treated as contradicting the clock, MUST be re-anchored to the current instant with a fresh lifetime before any expiry is evaluated (and the client hold re-granted its own maximum), so a reservation is never held beyond the maximum hold measured from an instant the node has observed.

Version in Cargo.toml is 0.252.31, unchanged as instructed.

Deliberately NOT done -- reported, not fixed

  1. Forward-jump early release. prune_reservations frees LIVE inputs when the clock jumps forward: the DELETE fires on expires_at <= now with no lower bound, so a forward jump past a live deadline retires a bundle that may still be in a mempool. On main today, unchanged by this fix, and a separate defect in the sec(reservation): gated on an untrusted 'accepted' — the under-claim direction fails OPEN into the double-select window #348/fix(wallet): a peer-local refusal must not free inputs another destination may hold #497 double-select direction.
  2. Reservation lifetimes are on wall-clock custody::now_ms(). That is the actual root cause of both this ticket and (1): a monotonic anchor for elapsed time, with wall-clock kept only for display, would make both unreachable. Much larger change; not started.
  3. reserved_coin_ids' clockless raw read -- over-reserves by design, safe direction, untouched.
  4. The sawtooth and timelock limitations on MAX_RESERVATION_HOLD_MS' doc -- documented, intended, unaffected.

Not merged, not undrafted, no review requested -- left DRAFT for the gate round.

MichaelTaylor3d and others added 14 commits September 2, 2026 05:57
Salvage anchor for the dig-node#502 lane. Version assigned 0.251.0
(origin/main is 0.247.0; 0.248-0.250 are held by sibling lanes).

Co-Authored-By: Claude <noreply@anthropic.com>
Four failing db-layer tests for dig-node#502, plus the constant they measure
against. `MAX_RESERVATION_HOLD_MS` is defined as a multiple of
`RESERVATION_TTL_MS` in one place so the two cannot drift; the TTL itself is
unchanged.

The acceptance test steps by less than a TTL past the cap and asserts its own
iteration count: a one- or two-push fixture is satisfied by the unfixed code,
because the first hold has not lapsed yet.

Co-Authored-By: Claude <noreply@anthropic.com>
…nputs

`reserve_spend` re-armed `expires_at` to `now + RESERVATION_TTL_MS` on every
push of a given transaction id, so a caller re-pushing the same signed bundle
more often than the TTL renewed its hold forever and the inputs never returned.
That is the lockout failure the TTL's own doc names as the worse of the two,
reachable without a single dishonest answer.

Two composed bounds, neither of which shortens the TTL:

* A TOTAL cap anchored on the FIRST push. `MAX_RESERVATION_HOLD_MS` is defined
  as `6 * RESERVATION_TTL_MS` in one place, so lengthening the TTL scales the
  cap and the two cannot drift. `submitted_at` is not in the upsert's
  `DO UPDATE SET` list, so the stored value is a stable anchor; a test pins it.
* A reason-conditional re-arm. `chain::refusal_forecloses_a_later_push` names
  the four CLVM-execution / cost refusals that complain about the bundle's own
  contents and that no better-synced node can turn into an acceptance. Those
  still HOLD -- the verdict is height-dependent, so this crate declines to trust
  one node's view of it -- but they may not RENEW the hold. Everything else
  extends, including an unrecognised reason, an `Err`, and a bare verdict.

The two compose into the gate's "every observed refusal was foreclosing" case
without a per-attempt history, because a re-push may never move the deadline
EARLIER: if every attempt is non-extending the deadline never leaves the first
`submitted_at + TTL`, and an extending attempt's grant survives every later one.

The clamp lives in the SQL so it is atomic against the stored anchor; a
read-then-write above this layer would race two concurrent pushes.
`coin_reservations`' `ON CONFLICT(coin_id) DO NOTHING` first-claim-wins rule is
untouched and pinned by a test.

Closes #502

Co-Authored-By: Claude <noreply@anthropic.com>
…clamp

The adversarial gate on #505 refuted the reason-conditional half and it is
removed in full: VIEW_DEPENDENT_BUNDLE_CONTENT_REFUSALS,
refusal_forecloses_a_later_push, attempt_may_extend_the_hold, the
PendingTransactionRow::may_extend_expiry field and the CASE arm in
reserve_spend.

The four names it listed -- GENERATOR_RUNTIME_ERROR, BLOCK_COST_EXCEEDS_MAX,
INVALID_BLOCK_COST, INVALID_SPEND_BUNDLE -- do not identify a bundle no
destination is holding. push_tx relays to up to three destinations and only the
LAST answer returns, so such a refusal from the last says nothing about the
first, which may have admitted and gossiped the bundle. The removed code freed
inputs up to 550s earlier than main for a bundle that lands, with no attacker:
the double-spend direction #497 exists to close.

What ships is the clamp alone: expires_at is bounded by
submitted_at + 6 * RESERVATION_TTL_MS. The outer MAX is retained because a
non-monotonic clock is the one case that can still drive an incoming deadline
below a live one, and shortening a live hold is the dangerous direction.

SPEC.md 18.9a gains the total-hold bound as a normative clause: without it a
reimplementation built from the spec as written reproduces the unbounded re-arm
this change fixes.

Three limitations are now stated in MAX_RESERVATION_HOLD_MS' doc and two are
pinned by tests: the bound is on CONTINUOUS hold and a re-push after the prune
gets a fresh anchor; a bundle whose timelock matures past the cap has its inputs
freed while the network genuinely still holds it; and inside the last TTL before
the cap a re-push buys strictly less than a full TTL. The clamp also fails OPEN
under an absurd clock, since SQLite promotes integer overflow to REAL.

Refs #502

Co-Authored-By: Claude <noreply@anthropic.com>
…total-hold

# Conflicts:
#	Cargo.lock
#	Cargo.toml
…total-hold

# Conflicts:
#	Cargo.lock
#	Cargo.toml
cargo fmt wanted the multi-line call form at both boundary tests. Formatted
those two files only; the workspace-wide check is now clean at zero diffs.

Co-Authored-By: Claude <noreply@anthropic.com>
The comment sat on `reserved_coin_ids`, which IS assembled from a stored
table and has no `true`. It described `may_extend_expiry`, the bool this
branch removed in fce2358, and pointed at a field doc that no longer exists.

Also tighten SPEC 18.9a: a re-push does not unconditionally 'update the
deadline' -- at or past the cap, and under a backwards clock, it correctly
leaves the deadline unchanged. State the re-arm as subject to the bound.

Co-Authored-By: Claude <noreply@anthropic.com>
…total-hold

# Conflicts:
#	Cargo.lock
#	Cargo.toml
…total-hold

# Conflicts:
#	Cargo.lock
#	Cargo.toml
…tion freeze)

Version anchor only. The fix follows: a far-forward clock at a bundle's FIRST
push writes a `submitted_at` far in the future, and #505's outer `MAX` then pins
`expires_at` there permanently, so no later correct-clock push and no prune can
ever release the coins.

Refs #525

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d and others added 2 commits September 3, 2026 01:12
#525)

`reserve_pushed_bundle` reads the clock once and writes both `submitted_at`
and `expires_at` from that reading, so a single reading far in the future
stores a deadline decades out. Nothing could retire it: `prune_reservations`
deletes on `expires_at <= now`, which never arrives, and #502's upsert clamp
is `MAX(stored, ...)`, so a later push under a corrected clock leaves the
stored deadline alone. The coin was withheld from selection for ever and
`reset_chain_cache` refused while the row existed.

`prune_reservations` now repairs at OBSERVATION: a row whose deadline exceeds
`now + MAX_RESERVATION_HOLD_MS` contradicts its own columns against the clock
(an honest row satisfies `expires_at <= submitted_at + CAP` and
`submitted_at <= now`), so it is re-anchored to `now` and granted one fresh
`RESERVATION_TTL_MS`. `submitted_at` moves too, or #502's cap clause would
stop binding on that row for ever. The client hold table gets the same repair,
keyed on the SAME threshold so a five-minute backwards step cannot re-clamp a
healthy hold, and granted its own ceiling since its requested TTL is
unrecoverable. All four statements now share one write-first transaction.

`reset_coin_db` prunes first, like every other reservation-sensitive entry
point, so the refusal message telling a user to wait becomes true.

Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

loop-security interim — IN PROGRESS, not the verdict

Audited head: 2edd6241ca170aa2382d75f5a03e6610646cd502 (resolved from gh pr view 528 --json headRefOid, matches the brief). Read-only; no shared checkout mutated.

Q1 — clock provenance on every prune_reservations path: CLEAN

Traced all six production call sites. None takes the instant from a caller, a peer, or an RPC parameter.

site how now_ms is obtained verdict
sage/rpc.rs:875 (reservations_held) let now_ms = custody::now_ms() as i64 local node clock
sage/rpc.rs:893 (reserve_coins) let now_ms = custody::now_ms() as i64 local node clock
sage/rpc.rs:1205 (reset_coin_db, NEW) parameter now_ms: i64 — see below node clock at the only caller
sage/rpc.rs:2969 (get_pending_transactions) custody::now_ms() inline node clock
sage/rpc.rs:3236 (spendable_coins) custody::now_ms() inline node clock
sage/rpc.rs:3703 (CAT send selection) custody::now_ms() inline node clock

The new site is the only one worth argument, because reset_coin_db(&self, now_ms: i64) is pub and takes the instant as an argument. Its sole production caller is dig-node-service/src/control.rs:2612, and that handler computes the instant itself at control.rs:2603-2609 from SystemTime::now().duration_since(UNIX_EPOCH) — never from params. The reasoning the brief asked me to check for is present and explicit at control.rs:2600-2602:

The node's own clock. A caller-supplied instant would be a lapse oracle: a far-future value makes every live spend reservation read as expired, which is exactly the guard being asked to stand down.

So the lapse-oracle reasoning documented on held_reservations was applied on this path, including the new one. custody::now_ms() (sage/custody.rs:441) is a plain SystemTime::now() read with no caller input.

No remote reservation-release primitive exists via the clock. A caller cannot pass an instant that deletes holds, and cannot pass one that trips the repair.

Still working: repair weaponisation without the clock (Q2), DoS/index/row-bounds (Q3), transaction+SQLITE_BUSYUnavailable (Q4), saturation (Q5), the anchor-move vs #502 (Q6), reset_chain_cache ordering (Q7), and the submitted_at truthfulness question (Q8) — which is the one I expect to take longest.

@MichaelTaylor3d MichaelTaylor3d left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correctness gate — PASS

Head reviewed: 2edd6241ca170aa2382d75f5a03e6610646cd502 (resolved from gh pr view 528 --json headRefOid; matches the brief).
Diff reviewed: git diff 15341ee..HEADSPEC.md, crates/dig-wallet/src/sage/db.rs, crates/dig-wallet/src/sage/rpc.rs.

Measured myself: cargo test -p dig-wallet --lib -> 800 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out, exit 0. Matches the brief's head figure. Non-zero test count, so this is a real run and not a filtered-to-nothing green.

1. The soundness argument is TRUE

Every production writer of pending_transactions.submitted_at / .expires_at was enumerated:

  • INSERT arm, db.rs:2801 — the only production caller is reserve_pushed_bundle (rpc.rs:2350, via reserve_spend; every other reserve_spend( hit in the crate is a test). It writes submitted_at = now_c and expires_at = now_c + RESERVATION_TTL_MS, so expires_at = submitted_at + TTL <= submitted_at + CAP.
  • DO UPDATE arm, db.rs:2805-2808MAX(stored.expires_at, MIN(excluded.expires_at, stored.submitted_at + CAP)). submitted_at is not in SET, and both MAX operands are <= submitted_at + CAP by induction. Invariant preserved.
  • The new repair UPDATE, db.rs:2924 — writes submitted_at = now, expires_at = now + TTL. Invariant preserved, and idempotent: a repaired row can never re-trip the detector.

No other statement in the crate writes either column. So expires_at <= submitted_at + CAP holds for every reachable row, and expires_at > now + CAP implies submitted_at > now. The doc comment's claim is correct. No false-positive class from a healthy writer.

2. The 50-minute floor is CONFIRMED (the shorthand formula is conservative, not wrong)

With E = expires_at - submitted_at = min(e_p + TTL, CAP), where e_p is the row age at its last push, the detector fires iff now < submitted_at + E - CAP. For a backwards step d on a row of true age e, that is exactly

d > e + max(CAP - e_p - TTL, 0)
  • Never-re-pushed row (e_p = 0, E = TTL): threshold e + 50 min.
  • At-cap row (E = CAP): threshold e, and e >= e_p >= CAP - TTL = 50 min.

The infimum over all reachable rows is CAP - TTL = 50 minutes, attained as e -> 0. The load-bearing number in the money argument is correct, and the earlier draft's "1 ms step trips an at-cap row" is refuted: an at-cap row needs d > e >= 50 min.

The comment's shorthand d > max(CAP - TTL, e) is a lower bound on the true threshold (exact in the at-cap case, understated for a young never-re-pushed row) — conservative in the safe direction. Non-gating; note (a) below.

3. The client-hold asymmetry is correct and correctly guarded

Healthy client row: expires_at_ms = now_w + ttl, with ttl clamped to <= CLIENT_RESERVATION_MAX_TTL_MS in reserve_client_coins. A detector at now + CAP therefore needs now_w > now + CAP - ttl >= now + 50 min — the same 50-minute floor as the bundle table, which is exactly the justification given. Keying on the table's own 10-minute ceiling would indeed have produced a ~5-minute false-fire floor.

The const _: () = assert!(CLIENT_RESERVATION_MAX_TTL_MS <= MAX_RESERVATION_HOLD_MS) at db.rs:4440 prevents precisely the unsoundness it names: raise the client ceiling above CAP and healthy client rows become detectable. The guard is load-bearing and correctly stated.

Granting the ceiling rather than the (unrecoverable) requested TTL errs long; for a build-window hold that is the safe direction, as argued. Post-repair expires = now + CLIENT_MAX <= now + CAP, so the repair is idempotent here too.

4. The transaction change is sound

  • Return value is still Ok(n) where n is rows_affected() of the first DELETE only (db.rs:2958-2969, :2991). The client DELETE result is discarded, unchanged. Read, not assumed.
  • A repaired row cannot be deleted by the DELETE below it: after repair expires_at = now + TTL > now, so expires_at <= ?now is false. It can still be deleted by the spent-coin cascade arm, which is correct.
  • Write-first ordering: self.pool.begin() is DEFERRED, and the first statement in the transaction is now an UPDATE, so the write lock is taken before anything else. prune_reservations contains no reads at all, so the read-then-write shape reserve_client_coins warns about at db.rs:4615-4619 is unreachable here.
  • The adjacent risk the brief did not raise: reserve_coins (rpc.rs:893) propagates prune's sqlx::Error through From<sqlx::Error> for ReserveClientCoinsError (db.rs:4567) into Unavailable. So prune's lock window does feed that mapping — but it did before this PR too (two autocommit writes), sqlx's default busy_timeout is 5 s, and four statements in one transaction is one lock acquisition instead of two. No regression.

5. The tests assert at the decision, and the corrected one is right

  • a_reservation_anchored_to_a_far_future_clock_stops_freezing_its_coin_for_ever asserts on unreserved_unspent_coins — the selection surface, not a raw row count. With the repair reverted, expires_at stays decades out, the DELETE never fires, and the second assertion goes red. Not vacuous.
  • Its start state (submitted_at = F, expires_at = F + TTL) is exactly what reserve_pushed_bundle writes from one clock reading. Reachable.
  • The boundary test's at-bound row (expires_at = submitted_at + CAP) is reachable via the DO UPDATE clamp, which leaves submitted_at alone and pins expires_at at submitted_at + CAP. past-bound is deliberately unreachable and is the negative probe. Correct pairing.
  • The #505 revert-proof correction (a_repush_under_a_clock_that_stepped_backwards_never_shortens_a_live_hold) is right: moving both columns by step_back is the only production-reachable shape, since reserve_pushed_bundle reads the clock once. step_back = 60_000 is far under the 50-minute floor, so the repair provably cannot fire and the test still exercises the clamp alone — its original assertion (expires_at unchanged) still holds for the now-reachable state.

6. Mutation claim — analytically confirmed, NOT executed

I did not run the mutation, and say so rather than implying otherwise: the tree is shared read-only and cutting a worktree for one compile was not worth the spend. Analytically, dropping SET submitted_at = ?now leaves submitted_at = F, so MIN(late + TTL, F + CAP) = late + TTL and MAX(corrected + TTL, late + TTL) = late + TTL, which is not corrected + MAX_RESERVATION_HOLD_MS — so ..._re_anchors_submitted_at_so_the_cap_still_binds goes red. The other five are unaffected: tests 1 and 2 depend only on expires_at, and tests 4-6 never touch that column. The unique-localisation claim holds.

7. SPEC.md

The insert lands at SPEC.md:5796-5808, inside the 18.9a In-flight coin reservation section opening at :5711. The second 18.9a at :5869 (offer suite) is pre-existing duplication and is untouched — nothing renumbered.

The normative text is true of the shipped code: repair before expiry evaluation (db.rs:2924 precedes :2958), client hold re-granted its own maximum (:2949), and the "never hold beyond MAX_RESERVATION_HOLD_MS from an observed instant" clause is discharged because the repaired row is anchored at now and the DO UPDATE cap binds from that anchor. No overclaim found.

8. rpc.rs:1198

reset_chain_cache's refusal counts pending_transactions WHERE expires_at > ?now (db.rs:3086). Pruning first repairs the poisoned row to now + TTL, which still refuses — but now refuses for a bounded time, so the message's "wait for them to confirm or expire" becomes true. No re-entrancy: prune commits its own transaction before reset_chain_cache opens one; no nesting, no lock held across the call. No ordering problem.

9. Beautiful-code gate

Comments are long but every paragraph carries a WHY, and I found no claim the code does not do. Names are intent-revealing; the test names state properties rather than outcomes.


Non-gating notes (recorded here, deliberately NOT as inline threads, so they cannot bar the merge)

(a) db.rs around 2900 — the false-fire shorthand d > max(CAP - TTL, e) is a lower bound, not the exact threshold; the exact one is d > e + max(CAP - e_p - TTL, 0). The stated 50-minute minimum is correct either way. Worth a word if the file is touched again; not worth a round trip.

(b) db.rs:2865-2869 — "A deferred transaction that read first would reintroduce exactly that" describes a shape prune_reservations cannot have, since it performs no reads. The ordering property asserted is true and the code satisfies it; the rationale is borrowed from reserve_client_coins, where it does bite.

(c) test ..._re_anchors_submitted_at_so_the_cap_still_binds passes far_forward as the re-push's submitted_at. It is inert (the DO UPDATE arm ignores that column) and the assertion is unaffected, but late would be the production-faithful value.

None of the three changes a behaviour, a bound, or a money outcome.

Verdict: PASS. No gating findings, so this gate leaves no open threads. The orchestrator owns the undraft and the merge; this review does not undraft, and the tree was not modified.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

loop-security interim 2 — IN PROGRESS, not the verdict

Head 2edd6241ca170aa2382d75f5a03e6610646cd502.

Q2 — can the repair be trip'd without touching the clock? NO. Clean.

Neither table lets a caller push a stored deadline past now + MAX_RESERVATION_HOLD_MS:

  • pending_transactions: the only production writer is reserve_pushed_bundle (sage/rpc.rs:2333-2351). It reads custody::now_ms() ONCE and writes submitted_at: now, expires_at: now + RESERVATION_TTL_MS. The caller supplies the bundle — hence transaction_id and reserved_coin_ids — but neither timestamp. Max reachable deadline now + 600_000 < now + 3_600_000.
  • client_coin_reservations: reserve_client_coins (sage/db.rs:4638-4642) takes a caller TTL but .clamp(1, CLIENT_RESERVATION_MAX_TTL_MS) it, so expires_at_ms <= now + 600_000. The new const _: () = assert!(CLIENT_RESERVATION_MAX_TTL_MS <= MAX_RESERVATION_HOLD_MS) at db.rs:4441 pins that relationship at compile time — a real guard, not vacuous.

The repair is also idempotent, which I checked because a re-firing repair would be a hold-renewal primitive. It grants now + 600_000 against a threshold of now + 3_600_000, so a repaired row cannot satisfy the predicate on the next prune. It fires once per row per clock excursion.

"Evade repair by staying just under" gains nothing: staying under the threshold is the healthy state, and the hold is still bounded by the expires_at <= now DELETE and #502's cap.


FINDING 1 — the repair DESTROYS a live hold on any backwards clock step over ~50 min, and the release is at clock RECOVERY, not "one TTL later". Severity MEDIUM, defense-in-depth-with-a-caveat — see the ask at the end.

sage/db.rs:2921-2929. The trade is disclosed in the comment at db.rs:2905-2919, and I agree with the direction of the decision — a permanent unrecoverable freeze is worse. But the disclosure's own sizing is wrong in the unsafe direction, and that is what I want on the record before this merges.

The comment says:

a backwards step exceeding 50 minutes can return a still-live bundle's inputs one TTL later instead of on clock recovery

Measured, that is backwards. The repair writes expires_at = now + TTL where now is the stepped-back clock, so the new deadline is in the PAST in real time. On clock recovery the very next prune's DELETE ... WHERE expires_at <= ? matches it immediately:

backwards step repair fires repaired expires_at deleted at clock recovery
0.5 h no unchanged no
0.9 h yes now-0.9h + 10min yes, immediately
5 h yes now-5h + 10min yes, immediately
13 h yes now-13h + 10min yes, immediately

So the real behaviour is "released on clock recovery", which is the phrase the comment uses for the safe alternative. A reader sizing this risk will read "one TTL" as ten minutes of remaining protection that does not exist.

Concrete scenario, no attacker required. pending_transactions is persistent across restarts. Node holds a live reservation for a bundle in the mempool → machine reboots → RTC reads hours off (dual-boot Windows/Linux localtime-vs-UTC RTC is the classic case, offset up to 13 h; a dead CMOS battery is another) → node starts, any wallet read prunes (spendable_coins, get_pending_transactions, reservations_held, reserve_coins, the CAT send path, and now reset_coin_db) → row is repaired → NTP corrects the clock → next prune DELETEs the row while the bundle is still live in the mempool → its inputs return to unreserved_unspent_coins → a second spend can be built on them. That is the #348/#497 direction.

Before this diff the same excursion was harmless: the row kept its true deadline through the excursion and expired on time.

Cheap mitigation that keeps the whole benefit. The detector's soundness argument only needs the threshold to be an upper bound no healthy row can exceed — MAX_RESERVATION_HOLD_MS is the smallest sound choice, which maximises false-fire exposure. #525's actual failure is a deadline decades out. A threshold well past any plausible clock excursion (e.g. now + 24 h, or an explicit CLOCK_CONTRADICTION_MARGIN_MS) still repairs the freeze this PR exists to fix while putting the false-fire floor above the RTC/timezone class entirely. It stays sound (strictly fewer fires), and the const assert just re-points at the new threshold.

I am not gating on this by itself — the decision to prefer a bounded release over a permanent freeze is defensible and is the ticket's whole point. What I do want before merge is the comment and SPEC.md corrected, because right now they understate a money-direction risk, and a future reader will size it from that sentence.

Still open: Q3 DoS quantification, Q4 SQLITE_BUSYUnavailable, Q5 saturation, Q7 reset ordering, Q8 submitted_at honesty.

@MichaelTaylor3d

Copy link
Copy Markdown
Contributor Author

Adversarial gate: NOT-REFUTED, with one finding that must be fixed before merge

Independent Opus context, prompted to refute. It could not construct a permanent freeze, and could
not find a cheaper early-release than the >50-minute backwards step already conceded on the record.
Suite independently re-measured at 2edd624: 800 passed, 0 failed, 1 ignored, 0 filtered out.

FINDING (gating, doc-only): the normative sentence this PR adds to SPEC.md is FALSE

"A node MUST therefore never hold a reservation beyond MAX_RESERVATION_HOLD_MS measured from an
instant the node has actually observed."

A forward clock glitch at first push evades the detector by construction, and the true bound is
2*CAP - TTL = 110 minutes, not 60.

To evade the detector at the first corrected observation a row needs expires_at <= now + CAP.
INSERT writes expires_at = submitted_at + TTL, so evasion holds exactly while the anchor is at
most CAP - TTL = 50 minutes ahead of true time. Take that maximum:

step clock reads call row
1 3_000_000 (glitch, +50 min; true time 0) first push submitted_at=3_000_000, expires_at=3_600_000
2 0 (corrected) prune_reservations(0) detector 3_600_000 > 0 + 3_600_000 is false - not repaired, not deleted
3 true t, rising a retrying caller re-pushes expires_at = t + TTL, so expires_at > t + CAP is never true
4 t = 6_000_000 cap clause binds expires_at stops at submitted_at + CAP = 6_600_000

Released at observed t = 6_600_000 = 110 minutes. The evasion window and the false-fire floor
are the same constant CAP - TTL seen from two sides.

The code is not wrong; the sentence is. This 110-minute ceiling is #505's cap behaviour under a
forward glitch and predates this PR. What this PR does is assert a tighter bound than either PR
delivers - a normative claim born false in the commit that writes it, which is exactly the class
#505's own body flagged when it noted the original defect was derivable from the spec.

Remedy: state the true bound and name the forward-glitch residue explicitly, the same way the
backwards-step cost is already on the record. No code change to the repair.

Two secondary findings, both accepted

  • Missing drift guard. There is a const _: () = assert! pinning CLIENT_RESERVATION_MAX_TTL_MS <= MAX_RESERVATION_HOLD_MS, but none pinning RESERVATION_TTL_MS <= MAX_RESERVATION_HOLD_MS. If
    the TTL ever exceeded the cap, a repaired row would satisfy the detector again on the next prune
    and re-anchor for ever - an unbounded freeze produced by the repair itself. Structurally
    impossible today (CAP = 6 * TTL), which is why it is a note and not a refutation; but if one
    constant pair is worth a compile-time guard, so is the pair that would turn the repair into a loop.
  • the_clock_contradiction_detector_fires_strictly_past_the_maximum_hold uses an unreachable
    row.
    Its past-bound fixture is submitted_at = now, expires_at = now + CAP + 1; production's
    INSERT always writes submitted_at + TTL and the clamp never exceeds submitted_at + CAP, so no
    writer can produce it. It is a predicate-boundary test wearing a production test's clothes - and a
    reachable equivalent (submitted_at = now + X, expires_at = submitted_at + TTL) would have
    surfaced the 110-minute finding above. The correctness gate read this row as reachable; it is not.

Attacks that failed, for the record

The induction survives an upgrade (both deadline columns are NOT NULL with no DEFAULT, there are
exactly two writers, no INSERT OR REPLACE, no migration, and pre-#505 rows satisfy
expires_at <= submitted_at + TTL anyway). The 50-minute floor re-derived without assuming the
span formula and holds in every branch, including the client table. No wire-reachable clock reaches
the repair: every prune_reservations caller takes custody::now_ms(), and the one parameterised
entry (reset_coin_db) is called only by control.rs, which reads the clock itself. No growing
fixed point in repair/re-push composition. No interleaving found that loses a hold or double-counts
rows_affected. A repaired row still counts toward ResetRefusal::SpendInFlight, so prune-first
cannot let a reset slip past a live hold.

MichaelTaylor3d and others added 6 commits September 3, 2026 07:06
…te forward-glitch residue

The adversarial gate on #525 found the normative sentence this PR added to SPEC.md was false:
it claimed a reservation is never held beyond MAX_RESERVATION_HOLD_MS (60 min) from an observed
instant. A forward clock glitch of up to CAP - TTL (50 min) at the first push evades the
clock-contradiction detector by construction, and the true worst case is 2*CAP - TTL = 110
minutes, tight.

- SPEC.md 18.9a now states the 110-minute bound explicitly, and names the CAP-TTL constant as
  both the forward-glitch evasion window and the backwards-step false-fire floor -- one constant,
  two sides.
- db.rs: doc comments on the repair explain the residue instead of overclaiming past it.
- A new compile-time assert pins RESERVATION_TTL_MS <= MAX_RESERVATION_HOLD_MS -- unreachable
  today (CAP = 6*TTL) but load-bearing if that ratio is ever narrowed, since a TTL above the cap
  would make every repaired row re-trigger the detector forever.
- The existing boundary test is renamed and its doc comment states plainly that its past-bound
  row is synthetic and unreachable by any writer -- it pins the SQL predicate's `>` only, not
  production behaviour.
- A new regression test, built entirely from real reserve_spend/prune_reservations calls (no
  hand-placed rows), measures the actual 110-minute residue under a real retry loop.

No behaviour change: the repair itself is unchanged. This corrects a normative claim born false
in the commit that wrote it, and pins the honest bound in its place.

Co-Authored-By: Claude <noreply@anthropic.com>
…-bound

# Conflicts:
#	Cargo.lock
#	Cargo.toml
#	SPEC.md
#	crates/dig-wallet/src/sage/db.rs
…-bound

# Conflicts:
#	Cargo.lock
#	Cargo.toml
…-bound

# Conflicts:
#	Cargo.lock
#	Cargo.toml
Co-Authored-By: Claude <noreply@anthropic.com>
…53.1)

Co-Authored-By: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Sep 3, 2026
Merge origin/main into loop/240-genesis-bringup and take the next
available patch version (0.253.0-0.253.2 are already claimed by
concurrent open PRs #524/#528/#518).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Sep 3, 2026
…l genesis (#240) (#533)

* test(peer): stub for genesis bring-up e2e (#240)

Co-Authored-By: Claude <noreply@anthropic.com>

* test(peer): prove bring-up installs the downstream engines on the real genesis (#240)

The acceptance test dig-node#240 has been open for: with the default DIG
mainnet genesis and no DIG_NETWORK_GENESIS override, the peer-network
bring-up gets past gossip-config validation and installs the pool handle,
the P2P content engine and the DHT inventory-refresh hook, then binds the
mTLS peer-RPC listener.

Asserts the DOWNSTREAM post-conditions rather than peerStatus.running,
which is set before GossipService::new and so holds even when the pool,
DHT, content engine and PEX all fail.

Also corrects a stale doc-comment that still described the genesis as a
pre-launch placeholder that invalidates the gossip config -- the reason
the #213 test sees no P2P convergence is environmental (relay off,
loopback only), not a rejected config.

Co-Authored-By: Claude <noreply@anthropic.com>

* style(peer): rustfmt the #240 bring-up test

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): v0.253.5

Merge origin/main into loop/240-genesis-bringup and take the next
available patch version (0.253.0-0.253.2 are already claimed by
concurrent open PRs #524/#528/#518).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
…54.3)

# Conflicts:
#	Cargo.lock
#	Cargo.toml
…54.13)

Co-Authored-By: Claude <noreply@anthropic.com>
@MichaelTaylor3d
MichaelTaylor3d marked this pull request as ready for review September 3, 2026 18:25
@MichaelTaylor3d
MichaelTaylor3d merged commit 4a90912 into main Sep 3, 2026
14 checks passed
@MichaelTaylor3d
MichaelTaylor3d deleted the loop/525-clock-anchor-bound branch September 3, 2026 18:25
MichaelTaylor3d added a commit that referenced this pull request Sep 3, 2026
#539)

* fix(wallet): discipline reservation liveness against a monotonic clock

Reservation deadlines (#502/#525/#528) are anchored entirely on wall-clock
readings. #528 closes the case where the clock is already wrong at the
moment a reservation is FIRST written. It does not close the general form
(#532): a wall clock stepped FORWARD while a reservation is already live,
mid-hold -- an NTP step, a VM pause/resume, an operator setting the clock --
produces no self-contradiction for #528's check to catch, yet the very next
prune reads the jump as elapsed time and can retire a bundle's hold while it
is still genuinely in flight, with no bound on how far forward the step
goes (the #348/#497 double-spend direction).

Add ClockGovernor: it disciplines every reservation-lifecycle "now" reading
so it cannot advance, between two observations, faster than a monotonic
clock says real time has actually elapsed. A forward wall-clock jump is
absorbed rather than trusted and the disciplined clock simply runs behind
until real time catches up, at which point it resumes tracking the wall
clock with no special unfreeze step. A backward step is passed straight
through unclamped, since it can only lengthen a hold, never shorten one --
the safe direction #502/#528 already accept elsewhere.

The governor lives for the process's lifetime and is not persisted: a
restart re-seeds it from the wall clock at that moment, so a clock already
wrong at boot remains #528's write-time contradiction check's problem, not
this one's.

Closes #532

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): bump workspace version to 0.254.20

Root workspace version, per the main lane -- the minor field's version scheme
is being fixed separately under #521/#522; this is the interim number to
carry PR #539 (dig-node#532) through the version-increment gate.

Cargo.lock refreshed in the same commit (cargo update -w --offline) so
dig-node-service's locked entry matches -- every CI job runs --locked, and a
manifest-only bump here fails Clippy/Test+coverage/all three package builds
together on a change that cannot otherwise break a build.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): bump workspace version to 0.254.43

Per the main lane: main advanced to exactly 0.254.41 after #535's rebase,
tying this branch's version. 0.254.43 clears main and every sibling PR in
the version-bump queue (#542=0.254.42, #543=0.254.44, #536=0.254.50,
#544=0.254.51).

Cargo.lock re-synced with `git checkout origin/main -- Cargo.lock` followed
by `cargo update -w --offline` (never hand-editing lock conflict markers),
confirmed clean with `--dry-run` -> `Locking 0 packages`.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
MichaelTaylor3d added a commit that referenced this pull request Sep 3, 2026
…543)

* fix(wallet): discipline reservation liveness against a monotonic clock

Reservation deadlines (#502/#525/#528) are anchored entirely on wall-clock
readings. #528 closes the case where the clock is already wrong at the
moment a reservation is FIRST written. It does not close the general form
(#532): a wall clock stepped FORWARD while a reservation is already live,
mid-hold -- an NTP step, a VM pause/resume, an operator setting the clock --
produces no self-contradiction for #528's check to catch, yet the very next
prune reads the jump as elapsed time and can retire a bundle's hold while it
is still genuinely in flight, with no bound on how far forward the step
goes (the #348/#497 double-spend direction).

Add ClockGovernor: it disciplines every reservation-lifecycle "now" reading
so it cannot advance, between two observations, faster than a monotonic
clock says real time has actually elapsed. A forward wall-clock jump is
absorbed rather than trusted and the disciplined clock simply runs behind
until real time catches up, at which point it resumes tracking the wall
clock with no special unfreeze step. A backward step is passed straight
through unclamped, since it can only lengthen a hold, never shorten one --
the safe direction #502/#528 already accept elsewhere.

The governor lives for the process's lifetime and is not persisted: a
restart re-seeds it from the wall clock at that moment, so a clock already
wrong at boot remains #528's write-time contradiction check's problem, not
this one's.

Closes #532

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): bump workspace version to 0.254.20

Root workspace version, per the main lane -- the minor field's version scheme
is being fixed separately under #521/#522; this is the interim number to
carry PR #539 (dig-node#532) through the version-increment gate.

Cargo.lock refreshed in the same commit (cargo update -w --offline) so
dig-node-service's locked entry matches -- every CI job runs --locked, and a
manifest-only bump here fails Clippy/Test+coverage/all three package builds
together on a change that cannot otherwise break a build.

Co-Authored-By: Claude <noreply@anthropic.com>

* fix(wallet): route wallet_reset_coin_db's now through ClockGovernor

wallet_reset_coin_db read its now_ms from a fresh, undisciplined
SystemTime::now() rather than WalletBackend::reservation_now_ms(), so a
wall-clock jump mid-hold (an NTP correction, a VM pause/resume) could make
its in-flight-spend check see a still-live reservation as already expired
and let the reset proceed -- the #348/#497 double-spend direction, no
attacker required.

reservation_now_ms() is now pub so the control plane (a different crate)
can route through it, sharing the same ClockGovernor clamp state every
other reservation call site (reserve_coins, prune_reservations) already
uses.

Swept every reservation-touching path in dig-wallet and dig-node-service
for a direct SystemTime::now() read; this was the only production one.

Closes #541

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): bump dig-node 0.254.42 / dig-wallet 0.49.0

dig-wallet: minor -- reservation_now_ms is now a public API surface
(dig-node-service routes through it, dig-node#541).
dig-node: patch -- behaviour fix, no breaking change.

Co-Authored-By: Claude <noreply@anthropic.com>

* chore(release): re-bump to 0.254.44 -- coordinator-assigned to avoid collision with #542

#542 keeps 0.254.42 (urgent required-CI-gate PR, merges first); 0.254.43 is
reserved for #539, which this branch sits on top of. Version assignment
across concurrent PRs is the coordinator's per CLAUDE.md section 1.4.

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A far-forward clock at first push freezes a reservation permanently: the outer MAX removes the self-heal a corrected clock used to provide

1 participant