perf(collateral): bound the cold-start census read budget by seeding the height search - #509
Conversation
Salvage anchor for the dig-node#404 lane.
Counts block_timestamp reads through the real dig_mirror_coin::census_height. For the SAME 20 epochs the search costs 504 reads at mainnet's peak, 544 at four times it and 584 at sixteen times it -- so a cold start is O(epochs x log peak) with both factors growing, which is the unbounded growth #404 is about. The budget test is RED until the seeded search lands.
…e census (#404) prior_record now returns the StoredRecord rather than the bare EpochRecord it wraps. The consensus record is still what advance() consumes; the envelope also carries census_height, which is a strict lower bound for the epoch being computed and is the seed the search needs. Behaviour-preserving: the line was already fetched and parsed, so this costs no extra read.
#404) The fixture stamps blocks uniformly and makes every block a transaction block, so it is the best case for an interpolated search and the worst case for reading its margin as a mainnet cost. Say so where the constant is defined, so nobody tightens it trying to measure something this fixture cannot see.
Co-Authored-By: Claude <noreply@anthropic.com>
Loop-security round-3 residual from dig-mirror-coin#11 (comment 5516011372, item 4)For any accepted seed S, one lying read at S prunes everything below it in the census search. This converts the adversary's required read count from "~log(peak)" down to "one". Not closable by a height filter alone — a liar returns height=peak and passes every downstream check. Fold in for this consumer: corroborate the seed-verification read across the cohort (multiple seeds or peers), or pass Version noteThis PR must now target dig-mirror-coin 0.9.0, not 0.8.0. PR dig-mirror-coin#10 merges first at 0.8.0, and #11 follows at 0.9.0. |
Lane ownership — orchestrator e93b41 (2026-09-02 21:35Z)Taking #509. Guard checked: no ownership claim newer than 21:00Z on this PR. Worktree: Plan: merge |
Resolves the two conflicts the merge raised: * `Cargo.toml` workspace version -- main carries 0.252.3 and this branch carried 0.253.0, which dig-node#501 now holds. Takes 0.254.0: a `perf` is a minor here. * `Cargo.lock` -- taken from main, then re-resolved for this branch's own dependency change in the commit that makes it. Brings in dig-node#506's `CorroboratedChainSource`, which the seed-trust decision in the next commit reasons about. Co-Authored-By: Claude <noreply@anthropic.com>
Progress — seed-trust decision settled, and one dmc 0.9 break folded inThe seed-read finding (20:43Z comment): corroboration is UNAVAILABLE, so the gate is on the seed's ORIGINI checked #506's What I did instead closes the part of the hazard that is actually a NEW trust dependency, and names the rest:
Discriminating test: One out-of-lane compile break, taken deliberatelydmc 0.9 adds a required Off-path finding, logged not fixed (§2.6): |
WIP salvage of the prior lane's uncommitted work: adopt dig-mirror-coin 0.9.0's seeded census-height entry point so the cold-start read budget is bounded by a caller-supplied lower bound instead of scanning from genesis. Co-Authored-By: Claude <noreply@anthropic.com>
…replaces (#404) The growth assertion used `census_height_seeded(.., None)` as its "before" baseline and measured 100 reads at every chain height, seeded and unseeded alike. dig-mirror-coin 0.9's unseeded fallback is an INTERPOLATED [0, peak] search, not the bisection dig-node#404 was filed against, so that baseline was measuring the fix's own worst case and reporting it as "there was never a defect". Split the policy into the three behaviours that actually exist -- `Bisecting` (`census_height`), `SeededWithNothing`, and `FromPredecessor` -- and point the growth assertion at `Bisecting`. Adds the hostile-seed case: a seed one block above the true census height, and one a whole epoch above it, must both be discarded and the search must still return the true height, with an honest seed as the truthful control so the assertion is about the seed being wrong rather than about seeds being ignored. The seeded-vs-unseeded comparison is `<=` deliberately: this fixture stamps blocks perfectly uniformly, so a strict `<` would assert a property of the fixture's linearity. The strict `<` is asserted against `Bisecting`, where it is a property of the change. Also repairs two assertion messages whose line continuations had been mangled into literal two-character `\n` sequences. Co-Authored-By: Claude <noreply@anthropic.com>
…start # Conflicts: # Cargo.lock # Cargo.toml
…rong for a fix Two defects in the version this branch claimed. It was UNBUILDABLE. All four package jobs failed identically at "Resolve + validate the package version": package-version: minor version 258 exceeds the MSI ProductVersion limit of 255 scripts/package-version.sh caps MAJOR and MINOR at 255 and PATCH at 65535, because Windows Installer's ProductVersion either rejects an out-of-range field or silently truncates it -- which would make two versions compare EQUAL. The check is deliberate and its own comment anticipates this exact case: "a stable 0.256.0 is just as unbuildable as a nightly one." It was also the WRONG BUMP. This branch is a `fix`, and CLAUDE.md 2.4 maps fix->patch, feat->minor. A minor bump was never owed here. 0.252.5 is a patch off main's 0.252.4: legal under the ceiling, correct for a fix, and it does not consume one of the three remaining legal minor slots (0.253/0.254/0.255), which open PRs #501/#509/#514 already hold. Cargo.lock line 3034 (dig-node-service) moves with it. The other 0.258.0 entries in the lock are upstream wasm-encoder and wasmparser and are deliberately left alone. Refs #508 Co-Authored-By: Claude <noreply@anthropic.com>
…his node's absence claim (#516) * chore(release): claim 0.258.0 for the forwarded-ask absence fix Salvage anchor for dig-node#508. Version claimed early so a concurrent lane does not compute the same slot; the fix follows on this branch. Co-Authored-By: Claude <noreply@anthropic.com> * test(download): red tests for dig-node#508 -- a peer's absence claim moves this node's verdict Two failing tests, both at the decision layer (`LocatedHolders::establishes_absence`): * a hop answering `absence_established: true` turns this node's inconclusive search into a proven absence, which `Node::availability_answer` then re-emits downstream at full strength. * an `Answered` whose records are ALL removed by the self-filter also arrives as a proven absence -- a second route into the same defect that does not touch `absence_established` at all. Co-Authored-By: Claude <noreply@anthropic.com> * fix(download): a forwarded ask never establishes an absence (#508) A peer answering `absence_established: true` for a subtree search that never completed moved this node from inconclusive to proven-absent, and `Node::availability_answer` then re-emitted `absence_established: true` to the next hop -- so an honest node laundered the lie and it travelled at full strength. The lie was free: an empty `Answered` and an empty `AnsweredInconclusive` are scored identically by both `ask_routing` and `conduct`. `ForwardedAnswers::asked()` now starts at `conclusive: false`, permanently. The reason is that ABSENCE HAS NO WITNESS: content from a stranger is safe to accept because the merkle root verifies it, and there is no verifier for "nobody has it", so a node may establish an absence only from its own completed search. Corroboration was considered and rejected -- `decide_forward` selects over the connected pool, which discovery and PEX can shape, and an eclipsed pool collapses any k-of-n to 1. `recursion_disabled()` is UNCHANGED at `conclusive: true`, which is what keeps every miss on every stock node a plain, provable not-found. This finishes #273 rather than extending it: that ticket closed silence-becomes-assertion (`unwrap_or(true)`); this closes stranger's-assertion-becomes-ours, the identical class with one door left open. Also closes a second route into the same defect, reachable today WITHOUT any `absence_established` claim: an `Answered` naming only this node is emptied by the self-filter after the flag was decided, so a peer manufactured a proven absence simply by answering "you hold it". SPEC: the three-state table's MEANING column is unchanged (it is the emitter's claim about its own search and stays true); only dig-node's READING column moves, and the resulting vacuity of the three states is stated explicitly. Closes #508 Co-Authored-By: Claude <noreply@anthropic.com> * chore(release): re-lock dig-node-service at 0.258.0 Co-Authored-By: Claude <noreply@anthropic.com> * fix(release): take 0.252.5 -- 0.258.0 is unbuildable and a minor is wrong for a fix Two defects in the version this branch claimed. It was UNBUILDABLE. All four package jobs failed identically at "Resolve + validate the package version": package-version: minor version 258 exceeds the MSI ProductVersion limit of 255 scripts/package-version.sh caps MAJOR and MINOR at 255 and PATCH at 65535, because Windows Installer's ProductVersion either rejects an out-of-range field or silently truncates it -- which would make two versions compare EQUAL. The check is deliberate and its own comment anticipates this exact case: "a stable 0.256.0 is just as unbuildable as a nightly one." It was also the WRONG BUMP. This branch is a `fix`, and CLAUDE.md 2.4 maps fix->patch, feat->minor. A minor bump was never owed here. 0.252.5 is a patch off main's 0.252.4: legal under the ceiling, correct for a fix, and it does not consume one of the three remaining legal minor slots (0.253/0.254/0.255), which open PRs #501/#509/#514 already hold. Cargo.lock line 3034 (dig-node-service) moves with it. The other 0.258.0 entries in the lock are upstream wasm-encoder and wasmparser and are deliberately left alone. Refs #508 Co-Authored-By: Claude <noreply@anthropic.com> * docs(download): correct a born-false test doc and pin the conduct inequality (#508) Two gating review findings on PR #516, both the same class: a claim asserted in a test that the code does not support. F1 - the doc on `a_holder_answer_whose_records_are_all_dropped_is_not_an_absence` claimed a "second route into the same defect [that] does not go through `absence_established` at all". False in the commit that wrote it. `parse_forwarded_answer` reaches `AskOutcome::Answered` by exactly two arms: `held && !records.is_empty()` (forwarded_ask.rs:351), whose leading `responder_record` names the responder and survives the self-filter; and `SubtreeClaim::Established` (:355), which by definition carries `absence_established: true`. The all-dropped state is production-reachable only as the original wire lie plus a `providers` entry naming us. The test is KEPT - the merge-layer property it pins is real. The doc now says what it actually pins (the merge-site self-filter, whose emptiness `establishes_absence` reads afterwards), states plainly that the `!establishes_absence` assertion is OVER-DETERMINED on this leg after the fix and is a regression guard rather than a measurement, and names where the same shape is genuinely live (the first-hand leg, download.rs:2201, held safe only by dig-dht's AddProvider caller check). F2 - `an_empty_answer_and_an_inconclusive_answer_are_indistinguishable_...` said `TimedOut` "must score differently in both dimensions" while asserting an inequality in the routing dimension only. A `conduct_evidence` collapsed to one value satisfied every conduct assertion. Adds the missing `assert_ne!(conduct_evidence(&answered), conduct_evidence(&timed_out))`; the two load-bearing equalities and the NonPerformance class assertion are unchanged. Docs and one assertion only - no production behaviour changes. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com>
…start # Conflicts: # Cargo.lock # Cargo.toml
…start # Conflicts: # Cargo.lock # Cargo.toml
…2.43) spends.rs: both sides added `declared_peer: None` for the dig-mirror-coin 0.9 bump; kept main's comment (cites dig-node#473). Code identical on both sides. Co-Authored-By: Claude <noreply@anthropic.com>
…2.80) Co-Authored-By: Claude <noreply@anthropic.com>
|
Merge-line lane (2026-09-03):
NEXT: CI on |
DO NOT MERGE - gate round pending.
Closes #404
Parent epic: https://github.com/DIG-Network/dig_ecosystem/issues/3173
Which of the three shapes this is
The ticket offered three: the walk is recomputed, the walk is over-wide, or the walk is not needed at cold start. Measured, it is the first — but not in the way the ticket guessed, and the other two are unavailable for reasons worth recording.
It is not recomputed across starts. Every epoch is computed exactly once;
EpochRecordStore::putrefuses to change history, andcatch_upre-censuses only the target epoch (#405). The redundancy is inside a single pass: each epoch re-bisects the whole chain to locate its census height, discarding everything the previous 102 searches established.dig_mirror_coin::census_height(census.rs:269) opens every search withlow = 0; high = peak. Nothing in the shipped stack memoises the probes —chia_query::provider_registry::ChiaQueryProvider::block_timestamp(chia_query_provider.rs:139) is a live round trip through a router that asksapi.coinset.orgfirst — so a cold start paysO(epochs x log peak)chain reads, with both factors growing over time. That is the super-linear half of the growth the ticket describes.Shape 2 (bound the window) is structurally unavailable.
EpochRecord::advanceis a stateful integrator:multiplier_micros = step_multiplier(self.multiplier_micros, saturation)(dig-mirror-collateral-0.3.0/src/record.rs:161), andsignals_foradditionally consumes the predecessor'scensus.storesandrequired_per_store_dig_base_units. Epoch n's price is a function of the entire record chain back to genesis, so a window starting at epoch 90 with a fabricated multiplier yields a different price from every other node — a consensus fork on the money path. SPEC §24.8a'sMUST NOT skip forwardis the normative statement of exactly that, and this PR does not propose amending it.Shape 3 (defer) is what the node already does, and it produces no requirement:
requirement()answersUnknown { NotCensused }throughout the walk, honestly. Deferring cannot satisfy the acceptance bar, which asks for a reported requirement in bounded time.What the census is for, and what reads it
It produces the per-epoch record from which a mirror operator's collateral requirement is derived. Four readers:
collateral::requirement(collateral.rs:855)store.get(current_epoch). No history.mirror/bond_verify.rs:457dig.getCollateralEpoch(server.rs:2562)collateral_sync(§24.10 adoption)So nothing reads history in bulk. History is walked only because epoch n's record cannot be derived without epoch n-1's.
The measurement
tests/collateral_census_cold_start_bound.rscountsblock_timestampcalls through the realcensus_height, with a fixture in which every block is a transaction block — so it understates the shipped cost, which additionally walks down through non-transaction runs.For the same 20 epochs, as the chain gets taller:
Extrapolated to the ticket's 103 epochs, with real non-transaction runs roughly doubling each probe: ~4,500-5,000 reads, i.e. ~140 ms per read against the eleven minutes measured on mainnet. The height search is essentially the whole cost — the censuses themselves are 103 reads.
The fix
Seed the search with what the walk already knows.
catch_uphas just recorded epoch n-1, and every record persists its census height (StoredRecord::censused(record, counted.height())), so a strict lower bound for epoch n's height is in hand. Given that seed and the peak's timestamp (already read by the existing not-started check), the first probe can be estimated rather than taken at the midpoint of the whole chain.Validated in simulation against a jittered, mainnet-shaped chain, asserting the returned heights are identical to the shipped search's in every case:
[0, peak]This removes the
log peakfactor entirely. One epoch per week remains, and is inherent to the model.The trust question, answered
A seeded search takes a lower bound as untrusted input to a consensus computation. A seed above the true census height would return a too-high height and derive a requirement no other node agrees with — SPEC §8.1: "A one-block disagreement here is a fork."
So the seed is verified against the source, not trusted: the crate re-reads the seed height's own timestamp and requires it to be strictly below the epoch start, discarding the seed and running the full search when it is not. One extra read, and a bad seed fails closed onto the correct answer rather than failing quiet onto a plausible wrong one. No checkpoint, no new trust principal, and no stored artifact that could be tampered with — which is why this is not the checkpoint design the ticket contemplated.
Sequencing (release-first, §4.1) — SATISFIED
census_heightlives indig-mirror-coin, a published crate, so the family was two PRs and the dependency went first.census_height_seeded, its equivalence and read-budget tests, SPEC §8.1. Released as 0.9.0, live on crates.io.dig-mirror-coin = "0.9"and carries the recorded census height forward as the seed.The earlier note in this body that the PR "is red until 0.8.0 is on crates.io" is obsolete: the dependency shipped as 0.9.0 and is adopted here.
A correction worth recording: the first version of the test had a FALSE CONTROL
The growth assertion originally used
census_height_seeded(.., None)as its "before" baseline. It measured 100 reads at every chain height, seeded and unseeded alike, which reads as "there was never a defect" rather than as a broken control.The cause is that dmc 0.9's unseeded fallback is an interpolated
[0, peak]search, not the bisection #404 was filed against — the crate's own docs say so. So the baseline was measuring the fix's own worst case.The test now distinguishes three behaviours instead of two:
Bisecting(census_height, the real pre-fix control),SeededWithNothing(the fallback taken when the predecessor's height is not self-censused), andFromPredecessor(the shipped walk). The strictseeded < bisectingassertion is made againstBisecting, where it is a property of the change; the seeded-vs-interpolated comparison is<=, because this fixture stamps blocks perfectly uniformly and a strict<there would assert a property of the fixture's linearity.The seed's provenance — a narrower rule than the crate's own verification
census_height_seededalready treats its seed as untrusted and verifies it against the source, so a bad seed there costs work rather than correctness. This PR adds a narrower rule that verification alone cannot express, incollateral_census::seed_from:only a height this node censused ITSELF may seed a search. A record with
RecordProvenance::AdoptedFromPeerscarries a height supplied by a peer cohort — a second trust domain, independent of the chain source — while the seed's verification probe is a singleblock_timestampread the chain source alone answers. Accepting a peer-supplied seed would let a peer cohort and a stale or forked source combine to prune the true height from below, which neither can do alone.Bootstrap(epoch 1, taken at no height) also yieldsNone, so the first search of a cold start is correctly unseeded.This follows dig-node#506's rule that a money-relevant verdict must not rest on one uncorroborated chain read, and discharges it by restricting whose claim may be used rather than by inventing a second corroboration mechanism.
Named limitation, stated rather than claimed: the height search itself is not corroborated across the peer cohort, because the corroborated surface (
dig_wallet::sage::CorroboratedChainSource, #506) answers by coin id and returnsUnsupportedforblock_timestamp. Every probe comes from one source whether the search is seeded or not, so seeding reduces the reads inside the existing trust boundary and does not widen it. Recorded in SPEC.md as a limitation, not glossed.Blast radius (checked, per §2.0)
census_heighthad one caller in this repo —collateral_census.rs:378, insiderecord_one— reached from one place,server.rs:3107(the census timer). That call is the only production line changed.seed_fromis new and private tocollateral_census.mirror/spends.rs::build_creategainsdeclared_peer: None, a field dmc 0.9 introduced.Noneis byte-for-byte what this builder has always written; naming a peer would credit collateral to it, and the read side already maps a silent coin toBondVerdict::Unverified, neverBonded.catch_upis otherwise unchanged: the walk stays sequential, history is computed exactly once, and everyCensusStopreason is untouched.Verified by grep and direct read. gitnexus
impactwas NOT usable and was not relied on: the registered index for this repo is ~301 commits stale, and a stale index returns a false-safeimpactedCount: 0, risk: UNKNOWNrather than an error (dig_ecosystem#3188).Dependency hygiene (§2.4b)
dig-mirror-coin0.7 -> 0.9. The wholeCargo.lockdelta is that bump plus this crate's version line. dmc 0.9 resolves onchia-protocol 0.36.1/chia-bls 0.36.1/chia-puzzle-types 0.36.1— the same linedig-node-servicecompiles against — so no chia line was added and nothing is split across two lines at a public signature. No otherdig-*dep in this crate is behind.How verified
dig_mirror_coinentry points, not a reimplementation, with every block a transaction block — so the fixture understates the shipped cost and every bound is conservative.peak, never wall-clock, so twenty epochs say the same thing as the ticket's 103.>= seedsearch would swallow) and one a whole epoch above it must both be discarded, with an honest seed as the truthful control.seed_fromproves the provenance gate is load-bearing: a gate written asprior.census_heightalone passes every other test in the file.Bump rationale
0.253.0 -> 0.254.0, minor. A compatible behaviour change with no API break: the census reaches the same heights by fewer chain reads, and the new provenance rule narrows which stored heights may seed a search. No public signature is removed or changed. Not a patch, because the read behaviour on the money path is materially different and a dependency's major line moved under it.