[rexjz]

This is #495 ported to layerfs_v2_0_0 (03e4c541a), as an alternative to #630. #630 restores #495's original two commits. Those were written against main before schema version 2, so they conflict with 2.0 in four files, and their triggers name inode_value and binding_hint, which 2.0 no longer has. This PR has the same two commits and the same API, with the changes 2.0 needs (listed below). If you would rather keep #630, its branch can take these two commits as they are.

What

A new module, layerfs_persistence::replication, keeps a second copy of a Store up to date by moving only the rows that changed:

• mark_base(store, token) makes an offline Store the start of an export chain. A copy of the closed file is then a replica at the returned cursor.
• export_changes(store, since, bundle) writes every row changed after since into a new SQLite bundle file, from one read snapshot of the live Store. It takes no write lock and stops no writer.
• apply_changes(replica, bundle) installs bundles in order into the offline replica, one transaction each. The replica then holds exactly the live Store's rows and opens through the normal path.

A bundle is a local file. Nothing here moves it anywhere: the caller decides whether it goes to another disk, another machine or object storage. The design is the same as in #495 (chained states, BLAKE3 digest, refusal before any install, publish by hard link, checkpoint report). See that PR for the details. Its open questions still apply.

What changed for 2.0

1. Schema version 2 blocks (v0.1.4 phase 2: benchmark against v0.1.3 and repair severe regressions #91). The triggers and the replicated table list now log inode_value_block (key first_ordinal) and binding_hint_block (keys grp, seq) instead of inode_value and binding_hint. There are still 18 logged tables, plus pack_unit in the group-row layouts.
2. The Doctor's type check. schema_diagnostic::type_checks compares each column's stored values with the column's declared type. change_log's key columns are declared ANY, which a STRICT table allows. ANY names no storage class, so every Store failed store.database.types, and about 20 diagnostic tests failed. The check now skips ANY columns (3 lines). The round-trip test also runs the Doctor on the live Store and on the replica, and expects no failure.
3. schema.rs. 2.0 made scripts() visible to the diagnostic module. This PR keeps that, and adds the change log script for each layout as before. The open-time migration add_change_log is unchanged. A Store written by 2.0 before this PR gains the two tables at its first writable open, as with the lineage, merge and digest tables.
4. Docs. docs/architecture/engine/store/sqlite.md describes the two tables and has a short "Change export" section. Doc 84 keeps its paragraph.
5. The tests use SqliteHandles and the block tables.

The second commit (apply no longer logs the rows it installs) is unchanged. On 2.0, without that commit, the free-page test still finds 22 free pages.

Not changed

• No change to object formats, the pack layout, history semantics, the publication lane, the PostgreSQL backend, or any layerfs-api, SDK or CLI surface. Nothing outside the module's tests calls it yet.
• Cargo.lock is unchanged: blake3 =1.8.5 moves from the dev-dependencies to the dependencies of layerfs-persistence.

Tests

tests/change_export.rs has 9 tests:

• the replica round trip, now with the Doctor on both sides
• every pack layout
• eight changed bundles, each refused, leaving the replica unchanged
• a written replica and a changed bundle
• a reader blocking the checkpoint
• a locked replica
• a replica outside the Disposable profile
• an unmarked Store
• the free-page test

lineage_backfill.rs, old_format_open.rs and sqlite_publication.rs are updated for the two new tables.

Checked on trees without each 2.0 change: without the ANY skip, about 20 tests fail in diagnostic_inventory, diagnostic_schema, diagnostic_snapshot and store_diagnostic. Without the second commit, the free-page test finds 22 free pages.

The loop again, on 2.0

I reran the two-machine loop from #495 with the 2.0 CLI built from this branch. Machine A writes through layerfs and exports one bundle per round. The bundle moves through a shared folder, and machine B applies it. B was restored from A's MinIO backup at r0. After each round, all 23 Store tables on A and B are compared row by row, and the same CLI reads run on both machines.

Round	Writes on A (CLI)	Bundle	Rows / deletes	Tables equal	Reads equal
r1	one file apply: write, mkdir, mv, cp, rm, edit	41,984 B	31 / 0	23/23	12/12
r2	30 single-file Commits, publish	94,208 B	383 / 0	23/23	16/16
r3	500 + 200 files on two Branches, three-way merge	205,824 B	1,446 / 0	23/23	24/24
r4	64 MiB text + 16 MiB random	29,955,072 B	4,409 / 0	23/23	29/29
r5	second Project shop/web: init + write	157,696 B	290 / 0	23/23	29/31 (gap)
r6	delete a file and a folder, project remove shop/web	33,792 B	24 / 11	23/23	35/35
r7	4 parallel writers × 15 Commits	145,408 B	787 / 0	23/23	43/43
r9	one Commit + publish (r8 is the same, exported before r9)	33,792 B	23 / 0	23/23	49/49

• The 9 bundles total 30.7 MB, of which r4 is 30.0 MB. A whole-file push after each round would move about 1.0 GB (the Store was 91–122 MB). Export took 2.4–58.5 ms and apply took 2.5–79.3 ms per round.
• Bundles carry fewer rows than on 0b8ae8af5: r1 went from 137 to 31 rows, r3 from 3,735 to 1,446, and r5 from 3,260 to 290. As far as I can tell, this is because inode values and hints now move as blocks.
• A skipped bundle and a replayed bundle are refused (Cursor). A replica with its own Commit is refused (Diverged). In all three cases, the replica's 23 table digests did not change.
• Every write on A succeeded, including all 60 parallel writes in r7.
• The gap from feat(persistence): export a Store's changed rows as bundles for replicas #495 is unchanged. In r5, the bundle carries the new Project's rows, but B's CLI answers NoSuchProject. 2.0 keeps Project records in the home, and apply_changes changes only the Store file.

Loop setup

• Linux aarch64 Docker containers (Debian bookworm) running release builds with Rust 1.85.1. A small driver binary, not part of this PR, calls mark_base, export_changes, apply_changes and read_state.
• A and B each have their own LAYERFS_HOME on ext4. At r0, A ran backup create and B ran backup restore, through MinIO (the pinned image from minio_service.sh). The snapshot carries the change state, so B starts at A's cursor.
• The Project is a git archive of 0b8ae8af5: 5,631 files, 132 MB.
• After each round: a digest of every row in all 23 tables on A and on B, ordered by primary key; the same CLI reads on both machines (file read/stat/list/glob/grep, branch list/get, commit list), compared byte by byte; and B's read_state, checked against the after cursor of A's export.

Checks run

CI is off for pull requests into layerfs_v2_0_0, so I ran these locally. Setup: Linux aarch64 (Debian bookworm container on a macOS host), Rust 1.85.1, cargo-nextest 0.9.146. I used the commands from engine/AGENTS.md "Checks and completion", ci.yml and nightly.yml, with every test command under a timeout. I ran the same commands on this branch and on 03e4c541a.

Check	This branch	03e4c541a
cargo fmt --all -- --check	fails in 2 files this PR does not touch	same
check_product_boundary.py, with and without --skip-sdk	2 findings in a file this PR does not touch	same
check_fuser_integrity.py, tools unit tests	pass
clippy --locked --workspace --exclude layerfs-node --all-targets -D warnings	pass
clippy -p layerfs-persistence --features postgresql --all-targets -D warnings	pass
cargo test --locked --all-targets --no-run	pass
fast lane (ci_tests.py targets/filter fast)	1901 run, 1901 passed, 26 skipped	1892 run, 1892 passed, 26 skipped
nightly lane (ci_tests.py targets/filter nightly)	146 passed, 11 skipped
mount lane (nightly.yml: each binary in its own container with /dev/fuse)	45 binaries, 116 passed, 3 ignored	same
every layerfs-persistence target	180 passed, 2 skipped

• The fmt and boundary findings are in control/crates/layerfs-store/src/lease.rs and api/crates/layerfs-cli/tests/concurrent_reads.rs, from 14f62997b.
• The fast lane ran as a non-root user with TMPDIR on ext4, as on the runner. As root, with TMPDIR on the container's overlay, both trees fail the same 59 tests: the kernel guard refuses overlayfs, and the daemon refuses uid 0.

Not run: x86_64, macOS, the Durable profile and layerfs-node.

Production LOC: 159967 -> 160998 (delta +1031): engine 74419 -> 75450, all in layerfs-persistence 17267 -> 18298; core, control and api unchanged. Per commit: +1030 and +1. Counted with tools/production_loc.py --json (blob 35e4a6d601) on git archive extracts of 03e4c541a and of each commit's tree.

[yifanxuaaa]

Review of the 2.0 port, read against 00bdc77887 on 03e4c541a. This was a code reading only: I did not build the branch or run its tests. #630 still carries #495's original commits (they conflict with 2.0 and name inode_value and binding_hint), so this PR is the one to review, and #630 can be closed in its favour.

Findings, most serious first

1. A marked Store's change log grows without bound. After mark_base, every write to the 19 logged tables appends a change_log row, one per object in object_location. Nothing on the live Store ever removes them: the only reset is mark_base again, which breaks every existing replica (they then need a full copy). There is also no way to turn logging off, and a Store restored from a backup of a marked Store keeps logging. Neither architecture doc mentions this. It needs a trim up to the oldest cursor still in use, or an unmark call, or at least a documented limit.

2. No measurement of the cost on Commit. Every Store now carries 57 triggers, and each written row checks whether the Store is marked. When it is marked, the writer's transaction also inserts log rows. AGENTS §3 rule 9 asks for both speed and storage numbers. The PR times export and apply, but not Commit or Save with logging off and on.

3. Every Store's schema changes for an API nothing calls yet. The first writable open by this binary adds the two tables and the triggers, after which an older binary (for example one pinned in a VM image or a worker) may no longer open that Store. Nothing outside the module's tests calls the API, and the review on #495 asked to agree the order with #506 and #533 in the 2.0 roadmap first. Whether this lands in 2.0 now is a call for the maintainers.

4. Nothing stops mark_base or apply_changes on a Store a daemon is using. The docs say both are for offline Stores, but they open their own SQLite connection with no busy handler and do not take the Store's writer lock (backend/sqlite/lane/writer_admission.rs). Run against a Store a daemon is writing to, the daemon's Commit can fail with Busy, which that lock exists to prevent under the no-retry rule. They should take that lock, or refuse when they cannot get it.

5. A bundle does not check that both sides have the same schema. history_meta, which records the schema, is copied like any other row. Apply only checks that each bundled table exists, then inserts with SELECT *, matching columns by position. If a later additive migration runs on the live Store after it was marked, the next bundle overwrites the replica's recorded schema without bringing the new table, and the replica then fails validation at open. A schema hash in bundle_meta, compared before installing, would catch this.

6. The Project-records gap (already disclosed). 2.0 keeps Project records in the home, outside the Store file, so after a bundle that creates a Project the replica's CLI answers NoSuchProject. Replication works within existing Projects, but not for adding one.

Checked and correct

• Store totals on the replica come out right. Commit and layer rows are only inserted or deleted, never updated, and any change to them also updates store_totals on the live Store, so the bundle always carries the totals and apply installs them last.
• No foreign key uses ON DELETE CASCADE, so apply's delete-then-insert cannot remove other rows. Deferring foreign-key checks to commit is correct.
• The refusals for a skipped, replayed or diverged bundle look right.
• The second commit (pause logging while installing, then restore the marker) is correct.
• Skipping ANY columns in the Doctor's type check is a small, reasonable fix.