Skip to content

wallet: add bwatch-backed output and transaction tables#9298

Draft
sangbida wants to merge 18 commits into
ElementsProject:masterfrom
sangbida:sangbida/bwatch-wallet
Draft

wallet: add bwatch-backed output and transaction tables#9298
sangbida wants to merge 18 commits into
ElementsProject:masterfrom
sangbida:sangbida/bwatch-wallet

Conversation

@sangbida

@sangbida sangbida commented Jul 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Overview

This is the third PR in the bwatch series. It introduces two wallet tables:

  • our_outputs, replacing outputs
  • our_txs, replacing transactions

Existing wallet data is backfilled into the new tables during migration. The wallet then uses these tables as its primary data source for transaction and UTXO state.

Compatibility and bisectability

The legacy tables remain in place for at least one release so users can downgrade without requiring a rescan.

During this transition, writes are mirrored to both the new and legacy tables. This keeps every commit in the PR series bisectable and ensures that a database written by a newer binary remains usable by an older one. Reads, however, are routed through our_outputs and our_txs.

Much of the apparent duplication in this PR is intentional and will be removed once the compatibility window closes.

Wallet scriptpubkey watches

This PR also adds persistent scriptpubkey watches for wallet addresses.

At startup, the wallet registers watches for its generated address forms. When bwatch finds a payment to one of these scripts, the wallet:

  • records the transaction in our_txs;
  • records the received output in our_outputs;
  • installs an outpoint watch to detect when the output is spent; and
  • handles watch reverts during chain reorganizations.

Longer-term direction

A later PR will stop updating the chain-topology-owned legacy tables, including:

  • blocks
  • transactions
  • outputs
  • utxoset
  • tables whose foreign-key dependencies require blocks

@sangbida
sangbida force-pushed the sangbida/bwatch-wallet branch 5 times, most recently from 2d96dc1 to a36f443 Compare July 17, 2026 01:30
@sangbida sangbida changed the title Sangbida/bwatch wallet wallet: add bwatch-backed output and transaction tables Jul 17, 2026
sangbida and others added 18 commits July 17, 2026 11:25
@sangbida @cursoragent
wallet: add our_outputs + our_txs schema migrations
4cfeca6 
The next commits move wallet UTXO and tx tracking off chaintopology and
onto bwatch.  bwatch doesn't maintain a blocks table, but the legacy
utxoset, transactions and channeltxs tables all have FOREIGN KEY
references into blocks(height) (CASCADE / SET NULL), so we can't just
retarget the existing tables.

Instead, introduce parallel tables (our_outputs, our_txs) without the
blocks(height) FK.  The new bwatch-driven code writes only to these,
the legacy tables stay populated by the existing code path during this
release so downgrade still works, and a future release can drop them
once we're past the downgrade window.

Losing the FK also changes what NULL means.  In the legacy tables a
NULL blockheight was never written by hand: the ON DELETE SET NULL
trigger produced it when a reorg deleted the block row.  These tables
have no such trigger, so unconfirmed is stored as blockheight 0
(NOT NULL) instead, for three reasons:

- everything feeding these tables already speaks u32-with-0: watchman
  notifications carry blockheight as a required JSON number, and
  wallet_transaction_height() has always returned 0 for unconfirmed,
  so values bind straight through without a NULL/non-NULL branch at
  every read and write site;

- integer comparisons keep working: the unconfirmed->confirmed
  promotion is a single "WHERE blockheight < ?" (0 sorts below any
  real height) and reorg rollback is "SET blockheight = 0 WHERE
  blockheight >= ?", where a NULL row would match neither;

- it removes the footgun the legacy code warned about ("Note:
  blockheight=NULL is not the same as is NULL!"), where lookups had
  to branch between "= ?" and "IS NULL".

The same logic gives txindex 0 = unconfirmed/unknown (a *confirmed*
txindex of 0 means coinbase, which blockheight disambiguates) and
reserved_til 0 = not reserved.  NULL survives only where 0 is a real
value or genuinely ambiguous: spendheight (NULL = unspent),
channel_dbid, commitment_point.

Schema only here — wallet handlers that write into these tables and the
backfill from outputs/transactions land in subsequent commits.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: record bwatch-discovered wallet outputs
d436e61 
Add the wallet helpers and watch handler that turn a bwatch scriptpubkey
match into our_txs and our_outputs rows.  They validate the matching
output, notify invoice accounting, record confirmed deposits, and
install watches for later spends.

Unlike got_utxo(), this path does not write transaction_annotations:
nothing reads per-transaction annotations anymore, so only the legacy
scanner keeps populating them (for downgrade, like the other legacy
tables).

The watchman dispatch entry is wired in the following commit.  Keep this
path alongside got_utxo() and wallet_transaction_add(): the legacy
scanner must continue populating outputs and transactions for one release
so downgrades do not require a rescan.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
lightningd: dispatch bwatch wallet scriptpubkey watches
076b84d 
Register the wallet/spk owner prefix with watchman so scriptpubkey
matches and reverts reach the wallet handlers introduced in the previous
commit.  Document the owner suffix format alongside the dispatch entry.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: add migrate_backfill_bwatch_tables
a440eae 
Populate our_outputs/our_txs from the legacy outputs/transactions
tables so the bwatch wallet path starts with the same state as the
existing wallet.

Use outputs rather than utxoset because it already contains only
wallet-owned rows and carries wallet metadata like reservations and
channel-close info.

Downgrade only drops the new tables: later commits keep mirroring
writes into the legacy tables, so no copy-back migration is needed.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: handle bwatch wallet/utxo spend notifications
4f9ff9f 
When bwatch_got_utxo() records a wallet output, it arms an outpoint
watch owned by wallet/utxo/<txid>:<outnum>.  This adds the receiving
end of that watch: the dispatch entry plus the found/revert handlers.

On watch_found (a tx consumed the outpoint): mark the output spent in
our_outputs, store the spending tx in our_txs, and emit a withdrawal
coin movement.  On watch_revert (a reorg removed that tx): clear
spendheight so the UTXO is spendable again.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: handle reorgs for our_outputs/our_txs by hand
c77e637 
The legacy tables rely on their blocks(height) ON DELETE SET NULL
foreign keys to mark rows unconfirmed/unspent when a block is reorged
out.  our_outputs/our_txs deliberately carry no blocks FK (bwatch does
not maintain a blocks table), so block disconnect and rollback must
clear their blockheight/spendheight fields explicitly.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: mirror bwatch writes into legacy outputs table
93718d5 
The bwatch path writes wallet UTXOs to our_outputs while the legacy
`outputs` table is what a downgraded binary reads.  Mirror every
our_outputs write (insert, spend, unspend-on-reorg, delete-on-reorg)
into `outputs` so a downgrade for one release needs no copy-back or
rescan.  The mirroring stops in the release that removes chaintopology,
freezing all the legacy tables at the same height.

The wallet still reads from `outputs`; switching reads over to
our_outputs comes next.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: dual-write chaintopology UTXO changes into our_outputs
d95251f 
Before this commit:

  chaintopology ──────────────> outputs
  bwatch ─────────────────────> our_outputs
         └────────────────────> outputs (downgrade mirror)

  wallet reads ───────────────> outputs

After this commit:

  chaintopology ──────────────> our_outputs
                └─────────────> outputs (downgrade mirror)

  bwatch ─────────────────────> our_outputs
         └────────────────────> outputs (downgrade mirror)

  wallet still reads ─────────> outputs

Make each existing chaintopology writer update both representations of
its UTXO state:

- wallet_add_utxo and wallet_add_onchaind_utxo insert into both tables;
- db_set_utxo updates reservations in both tables;
- wallet_confirm_tx updates confirmation heights in both tables; and
- wallet_outpoint_spend marks outputs spent in both tables.

Keeping each pair of writes in the existing helper makes the downgrade
mirror explicit without duplicating SQL at its callers.  Reads remain on
legacy `outputs` until the following commit switches them to our_outputs.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: read UTXO state from our_outputs
da09722 
With every writer dual-writing since the previous two commits, flip
the readers: all UTXO queries (listfunds, coin selection, reservations,
onchaind close info) now come from our_outputs, and the legacy-only
read helpers (wallet_stmt2output, gather_utxos, db_get_unspent_utxos)
are deleted.

our_output_row_to_utxo replaces wallet_stmt2output, using channel_dbid
to discriminate HD outputs from channel-close outputs, and
wallet_get_spendable_utxos centralizes the unspent+unreserved query
that wallet_find_utxo and wallet_has_funds previously duplicated.

migrate_setup_coinmoves keeps reading the legacy table directly: that
migration runs at v25.09, before our_outputs exists.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
tests: query our_outputs in test_connection.py
d6b605e 
The wallet's live UTXO state now lives in our_outputs; point the raw-SQL
assertions at it (spent means spendheight IS NOT NULL, reserved means
reserved_til > 0).

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
tests: query our_outputs in test_misc.py
36e8eb1 
Same raw-SQL switch as test_connection.py: read the wallet's UTXO state
from our_outputs.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
tests: add our_tables backfill upgrade test
1a003c0 
Upgrade a pre-bwatch snapshot db and check the migration mirrored
outputs/transactions into our_outputs/our_txs, listfunds still reports
the old UTXOs, and the wallet can receive and withdraw new funds.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: route transaction reads/writes through our_txs
8862408 
Now that the bwatch wallet path records every relevant transaction in
our_txs, point the wallet's transaction readers at that table.

Since our_txs.blockheight is NOT NULL with 0 = unconfirmed, the legacy
NULL handling disappears: wallet_transactions_by_height no longer needs
its "IS NULL vs = ?" query split, and wallet_transaction_height reads
the column unconditionally.

wallet_transaction_add keeps dual-writing the legacy transactions
table: the close path still inserts into channeltxs, whose
transaction_id foreign key points at transactions(id), and
wallet_get_funding_spend joins it.  That legacy write can only go away
with channeltxs itself.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: rename b32script to scriptpubkey in PSBT change paths
ef4018e 
Change addresses are no longer bech32-only (p2tr is the default form),
so the old name was misleading.  Pure rename, no functional change;
also drop the unused txfilter.h include.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: make datastore helpers self-wrap a wallet transaction
50f1fdf 
The four wallet_datastore_{get,create,update,remove} helpers used to
require the caller to be inside a wallet transaction; otherwise the
underlying db_prepare_v2 fatals at db/utils.c:103 with "Attempting to
prepare a db_stmt outside of a transaction".

watchman persists its pending bwatch ops through these helpers from
plugin callbacks that run outside any transaction, so wrap one on
demand.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
lightningd: only run watchman under --experimental-bwatch
b4943ac 
The flag is registered by the bwatch plugin, not lightningd, so peek
at the parsed configvars.  Without it ld->watchman stays NULL and the
watchman_* entry points are no-ops, leaving chain_topology as the only
chain watcher: bwatch and the legacy path must not race each other.

The bwatch pytests opt in explicitly (with rescan=0) instead of
enabling bwatch globally.

Changelog-Experimental: wallet: Add bwatch-driven wallet transaction and UTXO tracking behind --experimental-bwatch.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
wallet: register scriptpubkey watches at startup
6abe5a5 
init_wallet_scriptpubkey_watches walks every HD key (BIP32 + BIP86) up
to {bip32,bip86}_max_index + keyscan_gap and arms a watch for each
form, so bwatch can report deposits from the very first block it
scans.  wallet_get_newindex does the same for fresh keys, keeping
coverage as the wallet grows.

The per-UTXO watch on unconfirmed change is now redundant (the
perennial per-key watch already covers that scriptpubkey), so drop it.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida @cursoragent
tests: update test_wallet.py raw-SQL queries for our_outputs
7ca1f9c 
The wallet no longer writes UTXO state to the legacy outputs table's
status/spend_height columns, so tests that peek at the database
directly must read our_outputs instead.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sangbida
sangbida force-pushed the sangbida/bwatch-wallet branch from a36f443 to 7ca1f9c Compare July 17, 2026 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@sangbida