Repository navigation
Use a single socket for gossip - #52
Merged
Merged
Conversation
`tools/digestshare` measured the share of wire bytes that subtree digests take, reading the committed wire captures. Its stream-header pattern matched the per-stream labels the previous commit removed, so it now counts no frame bytes, and its liveness check passes on control items alone: it can no longer fail for the reason it exists. Delete the tool, its `digestshare` recipe and that recipe's comments in the `justfile`, and its entry in `gate-lints`. Nothing replaces it.
The frame writer issued one write per piece of a frame: the opener, then the query listing or the supply run's head and records. Over TCP with Nagle's algorithm on, a short opener followed by its body is the write-write-read pattern that waits on the receiver's delayed acknowledgement, and a session makes that exchange at every dependent crossing of its descent. - `codec/encode.rs` and `codec/encode/async_io.rs`: a frame's pieces go out as one vectored write, continuing with the remainder when the transport takes only part of it. A write failure names the piece holding the first byte the transport did not take. - `stats.rs`: `CountedWrite` passes vectored writes through, so a TCP socket or a `BufWriter` takes a frame in one operation. - `lib.rs` (and the generated `README.md`): the transport docs tell TCP users to disable Nagle's algorithm on both ends, and why. The test kit's loopback TCP pair does (`common/tcp.rs`). - `codec/tests.rs`: a transport that takes vectored writes whole receives each frame as one write, byte for byte the synchronous encoding; one that fails partway, after vectored writes of any size, fails the frame naming the same piece the synchronous encoder does. Wire bytes are unchanged.
…y frames The frame writer flushed after every frame. Encoders publish their questions when a frame enters the multiplexer's channel, not when it is written, so the per-frame flush served no correctness purpose; it cost one syscall or TLS record per frame on a buffering or encrypting writer. `FrameWrite::frame` now writes without flushing, and `FrameWrite::flush` flushes only when a frame was written since the last flush, naming that frame if it fails (`codec/encode/async_io.rs`, `codec/error.rs`). Flushes stay where progress needs them: - the multiplexer flushes whenever no frame is ready, before it waits for one, so a frame handed to it never sits unflushed while it waits; - it flushes before returning, since nothing else writes until the closing exchange and the peer may need those frames first; - the preamble, the greeting, the completion marker, and a donated identity still flush as they are written. Docs: the crate's transport docs (and the generated `README.md`) say a buffering write half is safe and batches the messages written between flushes; the observation docs say a sent frame is observed once written whole to the write half, which may still be buffering it. Tests: the proxy tests' scripted writer splits bytes into CBOR items as they arrive and delivers items only on flush, so a session that waited without flushing stalls there; the deep proxy test sizes its flush-delay schedule by the baseline's writes, which bound the flushes under any schedule, since flush counts now vary with scheduling. The codec tests and error atlas follow the new flush error. Wire bytes are unchanged.
Tokio's `mpsc` aligns two fields of its shared state to the target's cache-line size (128 bytes on x86-64, AArch64, and powerpc64), so each channel is an over-aligned allocation, which the system allocator serves through its aligned-allocation entry point. That path is slow on some allocators, illumos's libumem (Rust's default allocator there) among them. A session creates its channels anew, so the cost would recur on every sync. With the link layer gone, the library uses none. - `clippy.toml` disallows Tokio's `mpsc` types and constructors (the types catch every written mention, the constructors a channel whose halves are never named), and explains why `broadcast`, `watch`, and `oneshot` stay allowed: none is over-aligned, and `watch` is created once per peer rather than per session. - The two remaining uses convert: the bookmark cancellation test queues jobs on an unbounded `futures-channel` (`bookmark/file/tests.rs`), and the quiescence test spends Tokio's cooperative budget directly with `coop::consume_budget` (`testing/tests.rs`). - `channel/bounded.rs`'s module docs state the same cost, in place of a claim that libumem serializes aligned allocations behind one lock.
The walk and the proxy each resolved the session window from their own `WindowConfig`, so every caller passed one configuration twice. Parking's capacity rests on the proxy's window equaling the walk's (both are sized from the count of questions this side has outstanding), and nothing but the callers' discipline kept the two copies equal. The proxy now holds no window configuration: - `protocol.rs`, `protocol/peer.rs`, `streaming.rs`: a `ShareWindow` phase trait, part of every participant's phase chain, lets `descend` take the window each participant resolves and hand it to the other once the greetings' versions differ. The walk resolves its window and ignores its counterpart's; the proxy adopts the walk's. Two walks in one in-process session each keep their own. - `materialized.rs`: the walk resolves its window from both greetings lazily and caches it, so an equal-version session still solves no window and reports none granted, and each endpoint solves once instead of twice. - `proxy/start.rs` and `proxy/state.rs`: the proxy stops resolving its own window and takes the walk's. - `peer/gossip.rs`, the proxy tests' harness, and the streaming test drivers stop configuring the proxy's window; `window.rs`'s docs say the proxy adopts the walk's. Windows do not move.
The proxy's progress trace checked only that the takes under a label (endpoint, stage, and height) never outnumber the questions published before them. Once the encoder's and the take's labels agree, the decoder's pairing of replies with dequeued question records guarantees that already, so the check could not fail on what it can actually observe: an encoder whose runtime height label disagrees with the take's typed height, or a question published and never taken. `assert_question_causality` becomes `assert_takes_answer_published_questions` (`proxy/work/progress/trace.rs`). Besides the prefix inequality, it requires every label's takes to equal its published questions by the end of a completed session; the root opening, which answers the greeting, stays the one take without a question. Its docs say it holds only for completed sessions. The trace's own tests cover a take after its question, a take under a label with no question, another endpoint's question, and a question never taken; the minimal divergent session the floor test records now includes the responder's take of its answer. The proxy suites call the renamed check.
… itself Three assertions in the proxy suites held a queue's high-water mark to that same queue's capacity, which the bounded channel enforces by construction, so they could not fail. The channel's own tests check that contract directly. The three assertions go (`proxy/tests.rs`, `proxy/tests/deep.rs`). Where a capacity was the claim, the deep-queue test now asserts that the deep question queues take the window's width, which the old comparison assumed rather than checked.
`formal/MODEL.md` described the remote session as running over a caller-supplied link whose independent per-level streams the model's wire FIFOs stand for, validated by the link conformance suite. Since the previous commits a remote session runs over one ordered byte stream, and both the link and its suite are gone, so that bridge describes a transport that no longer exists. `MODEL.md` now says what holds for one byte stream: - kernel-checked at message grain: the elastic mux theorem (`elastic_deadlock_free`), with `wc_impossibility` as its contrast for one-slot or fixed-depth parking; - argued in the Rust, not kernel-checked: that the remote session is an instance of that elastic variant (the pipe module's table of waits on the receive path, and `parking_capacity`'s derivation), including the byte-grain framing below the model's grain; - not composed: no theorem covers the elastic pipe at the walk's window-widened capacities; - outside the model: the codec and the proxy's own queues, validated by the Rust suites. It also fixes one path (`materialized/work/levels.rs`). The mux charter's value statement (`Mux/Charters.lean`) drops the deleted link contract and states its claim in plain words, and `EventDag.lean`'s "experiment knob" becomes an experimental setting.
…l development Comments and test docs in the streaming tests and the walk's progress trace cited Lean theorem and definition names as the argument for their claims. The Lean development models the transport the previous commits removed, and the next commit archives it, so those citations would rest claims on an unmaintained artifact. Each invariant is now stated inline, naming the Rust check that holds it where one exists, and claiming only what the tests and the Rust arguments support: - `materialized/progress/trace.rs`: the validator's invariants are what the walk's progress relies on, and completed sessions passing it are empirical evidence, not a proof; the pipe module's docs give the argument for one pipe. Wire contiguity's supported role is as the premise of the in-flight count parking is sized by. - `materialized/progress/tests.rs`: publishing every resolution before any query deadlocks only when a stage publishes capacity plus two resolutions, the first owing work, before that work. - `streaming/tests/capacity.rs`: the parent fan boundary test states what it shows at capacities one and two. - `streaming/tests/dispute_before_supplies.rs` and `skeleton.rs`: the fixture shows that a receiver must hold supplies its stage has not yet taken, and that one-slot parking overflows on them; the stall a waiting reader would meet is argued from the code, and the shape is not claimed to be the smallest such. - `announced.rs`, `local_eq.rs`: their formal citations become plain statements. Test items that carried the formal name take names after what they are (`SHAPE_FAN`, `SHAPE_MAX_D_COUNT`, `height_six_literal`, `generator_matches_the_literal_at_height_six`). No behavior changes.
The Lean development is not maintained alongside the code: it models the stream-bundle transport, and after the previous commit no code relies on it for an argument. - `formal/README.md` and `formal/MODEL.md` open with a notice that the development is archived and not maintained alongside the code: its theorems are about its own model of the protocol, and the code does not rely on them. - `AGENTS.md` drops the rule that let code cite the Lean artifact by theorem name. - `justfile`: the local-only sweep, and with it `just all`, no longer runs the Lean recipes (`lean`, `eventdag`, `muxprobe`); they remain under the formal tier as standalone recipes, and the tier's comment and the header's `all` line say so.
Questions about throughput need session durations on a link whose bandwidth, propagation delay, and kernel send buffer all shape timing. The latency fixture models delay alone on Tokio's paused clock, whose millisecond grain is far coarser than one segment's serialization. `bench::link` (`crates/rumors-testkit/src/bench/link.rs`) drives two peers on a discrete-event clock counted in byte-times: bandwidth is one byte per byte-time, so a link is described by its one-way delay, and a session's duration is an exact function of its fixture and link, unmoved by machine load. Each direction models a sending kernel's unsent queue with a limit; a wire taking segments in order within an in-flight window of one bandwidth-delay product plus two segments; delivery one one-way delay after a segment's last byte; and acknowledgements as the reader takes bytes. The new `throughput` binary (`tests/throughput.rs`) holds the deterministic pairs and session runner the session suites share (`throughput/fixture.rs`) and checks the model itself (`throughput/model.rs`): its timing on traffic computed by hand, its critical path against the paused-clock link, and, with a reader that pauses until well after its sender has stalled while its own stream back keeps the clock moving, that the sender stops with exactly one product plus two segments in flight and starts again exactly one one-way delay after the reader takes them.
A ledger reading counts bytes, but it cannot say which of the blocks a region allocated are still live when the region ends, and it cannot attribute the work of several computations interleaved on one thread. Answering either needs the individual requests, in order. This adds the recording half of a trace. Inside `trace(capacity, f)`, the global allocator appends one event per request (the block's address and size, what the request did to it, and the current label) to a buffer reserved before `f` runs. `labelled` runs a closure under a label, and `untraced` omits the allocations a closure makes, for buffers a harness owns. A trace that runs out of room counts what it dropped instead of growing. - `trace.rs` (new) holds the thread-local mode and buffer, the recorders the allocator calls, and the public `Label`, `EventKind`, `Event`, and `Trace` with its events and drop count. - `lib.rs` exports the module and calls the recorders from each `GlobalAlloc` method. The safety comment says why recording never allocates or re-enters the allocator: the buffer is installed before tracing starts, and a push happens only into spare capacity. - The existing tests' `Block` helper becomes `pub(crate)`, so the new recording property can issue raw allocator requests and predict every event from the blocks it holds. The `trace` docs state what a reservation costs on a system without overcommit (illumos reserves backing store for the whole capacity at once), so callers size it near what the work needs. The next commit classifies a trace's blocks into lifetimes; nothing else uses the trace yet.
A recorded trace is a list of requests; a measurement needs to know, for each label, how many bytes the region held that it also freed, and what it left behind. `Trace::lifetimes` pairs each free with the most recent live allocation at its address and sorts every block the region allocated into one of two classes: *transient* (allocated and freed inside the region) or *surviving* (still live when it ended). Blocks from before the region, or allocated inside `untraced`, match no traced allocation and are ignored. For each label it reports the *transient peak*, the most transient bytes live at once, with a reallocation counting its old and new blocks together. `lifetimes_grouped` classifies as though each label were replaced by a group a function maps it to, so one trace yields both each label's peak and a group's peak across its labels. `survivors` lists the surviving blocks themselves. A trace that dropped events refuses to classify (`Incomplete`), since a lost free would turn a transient block into a surviving one. An allocation at an address a live block still holds ends that block, which another thread must have freed, and is counted in `Lifetimes::reused_while_live`. - `trace.rs` gains the analysis (`Lifetimes`, `LabelLifetimes`, `Incomplete`, and the pairing that both passes share) and the module docs' sections on classification and grouped labels. - `lib.rs` re-exports the new types and its crate docs point at the trace. - `trace/tests.rs` checks the classification against a model heap whose every block's fate is known, grouped and ungrouped, plus the reuse of a live address and a trace that fills its buffer.
A ledger kept beside a trace can be held to the transient heap at every allocation event only if the meter can say two things it lacked: where the trace stands when the ledger changes, and the most transient bytes a group of labels held between two such changes. - `position()` returns the index the running trace's next event will take (events recorded and dropped so far), or `None` outside a trace. It neither allocates nor panics, so the allocator's bookkeeping is undisturbed. - `Trace::segment_peaks(starts, groups, group)` returns, for each stretch of the trace between consecutive positions the caller names, the peak transient level of each group of labels and of every group together, in one replay. An event's level counts a group's transient blocks as the classification does: a block a reallocation moves stays live through its new block's allocation, and a reused address's block through the reusing allocation. A caller passing its account's change positions as the starts gets, per segment, the most each group's level reached against the one value the account held there. The analyses that take a caller's function over labels (`segment_peaks` and `lifetimes_grouped`) call it once per distinct label and replay through a loop that is not generic over it, so the replay compiles with alloc-meter rather than with each test binary that calls it. Tests: properties holding `segment_peaks` to an interval model of block lifetimes over raw event sequences that reuse live addresses and free blocks the trace never saw, and to the classification's transient peaks over the model heap; and a point test that the position counts the events recorded so far.
To compare a session's memory with an account of it place by place, a test must know which allocations serve which place: the walk's questions, the proxy's parked replies, the replies on their way to the wire, and so on. Nothing in a session said which place an allocation was for. The new `holding` module (`streaming/holding.rs`) names the places a session keeps in-flight state (`Holding`: questions, resolutions, question records, parked replies, decoded leaves, outgoing replies, wire buffering, assembly, and exploded nodes), each doc saying what the place holds. The session marks the code that allocates for each place: synchronous work through `held_by`, and a task or stream that allocates for one place throughout through `holding` and `holding_stream`. An allocation counts toward the place whose code made it, wherever the value later moves, so the marks sit where each place's contents are built: a question's child listing where the walk explodes the node it asks about (`materialized/work/answer.rs`, `levels.rs`), a parked reply where the decoder assembles it (`adapter/decode.rs`), wire frames in the encoders and decoders (`adapter/encode.rs`, `codec/decode.rs`, `proxy/work/encode.rs`, `proxy/work/stages.rs`), and the walk, park, take, and assembly tasks (`materialized/work.rs`, `proxy/work.rs`). Nodes a backend builds to explode a node are exploded nodes (`erased.rs`). Moving a value between places allocates nothing but a channel's per-message node, which belongs to the sending place: each channel edge maps to its place (`QueueKind::holding`, `channel.rs`), and the production sender carries it (`channel/bounded.rs`). The edges of fixed capacity, the assembly returns and the terminal leaf resolutions, belong to assembly. The marks call a hook a test installs once for its process (`testing::attribute_allocations`), inside which the test labels the allocator's records; the innermost mark applies. Without the `test-internals` feature every mark is the bare expression or future it wraps, and nothing here exists at run time.
…ns against it The ledger the following commits build prices each question by the reply it invites, so it needs an oracle for the replies a session builds that does not run the walk. This adds that oracle as a differential test, before anything it will judge changes. The descent model in tests/descent.rs computes every reply both walks build from each replica's leaf paths plus one bit per leaf, whether the peer's greeting version leaves it unknown, which is all the deletion filter reads. It runs over an explicit stack, so its depth is independent of its input. The suite runs sessions, in process and over a pipe, captures every reply each walk publishes through the existing transcript, and requires the two to agree per endpoint and logical stream: the same replies, in the same order, with the same reactions. Every pair runs at the one-slot floor and at a wide window, since a reply's contents depend only on the trees. Fixtures: the version-addressed divergent pairs (shared history, redaction), pairs placed below a shared prefix of drawn length so disputes reach every depth (the wire run derives leaf paths from the placement), and two 2,000-message forks with redaction in one fork. The transcript names endpoints by work identity, not role, so the check matches the two endpoints to the two predicted roles as an unordered pair. skeleton's Party gains an ordering for use as a map key, and the pipe suite's placed-session helper becomes visible to sibling suites.
The in-memory backend's assembly of a supplied subtree buffered each run whole as leaf-run entries (about 40 bytes a leaf) before building it, so its transient memory grew with the subtree, about n/256 leaves per root child when a peer joins from empty: the one place in a session whose size no count the protocol carries can bound. `Local::assemble` now builds each run one leaf at a time with a right-spine builder (`untyped::SortedLeaves`, typed `SortedRun`). It keeps at most one open branch per path byte, each with the children closed so far, plus the last leaf. Each push compares the leaf's path with the previous one's and closes every open branch deeper than their divergence byte, building it once with its whole compressed prefix. The result is exactly the node `from_sorted_leaves` builds: same prefixes, fans (fitted), leaf counts, and hashes. `from_sorted_leaves` remains, test only, as the reference the builder is tested against; `LeafRun` goes. The trait's default `assemble` (`Convert::assemble`, `fold_parents`) already streams, holding one group of at most a fan per height. Its docs now say so and ask an override to keep that bound. Choice, by deterministic allocation-meter counts per supplied leaf on ox-east-1 (release), for a subtree assembled at the root; the counts are allocations / bytes allocated / transient peak: leaves buffering override default fold (a) spine builder (b) 10^3 0.44 / 121 B / 61 KB 30.7 / 603 B / 24 KB 0.44 / 80 B / 3.4 KB 10^5 0.43 / 142 B / 7.9 MB 29.9 / 587 B / 29 KB 0.43 / 89 B / 5.5 KB 10^6 0.16 / 87 B / - 29.2 / 532 B / - 0.16 / 45 B / - All three build the same nodes (249 and 29,992 node allocations at 10^3 and 10^5). The spine builder makes about 70x fewer allocations and 6.6x fewer bytes than the default, so it ships; the probe that produced the table is not committed. Tests: - `spine_build_matches_the_bulk_build`: a proptest that the builder's tree equals `from_sorted_leaves`' at every starting depth, structurally (prefix bytes, hashes, leaf counts, radices, leaf identity by version), with fans fitted; plus the empty and repeated-path cases. - `tests/assembly_alloc.rs`: assembling 2, 1,000, and 100,000 leaves holds at most the one-leaf build's transient peak plus `ASSEMBLY_SPINE_BYTES`, the spine's bound, with a liveness floor of one surviving branch. The buffering override fails it at 100,000 leaves (7.9 MB). The test kit's hooks (`supplied_leaves`, `assemble_supplied`) build and assemble the leaves as a reply decoder hands them over. `Holding::Assembly` and the decoder's assembly note say what the backend now holds.
…regate The reply a question invites is bounded by the replier's grandchild count of the node it asks about, so every backend must answer that count exactly, as it answers len. A node's grandchild count G is, over its children as children_of lists them, each child's own child count, a child at the leaves counting one. A child's count depends only on its storage (one if its path is compressed or it is a bare leaf, its fan's length if it is a bare branch), so each branch body stores one aggregate, S, its count as a bare branch, as S - 1 in a u16 (`Body::Branch::spread`). A branch has 2 to 256 children each counting 1 to 256, so S lies in 2..=65,536 and the field is exact. Every other node's G follows from its prefix: two or more path bytes give 1, one byte over a branch gives the fan's length, one byte over a leaf gives 1, and a bare leaf has none. Each branch constructor computes S from its final fan in one pass: `Node::branch` (insert, join, redaction and pruning, the deletion filter's rebuilt branches, assembly's parent), `from_sorted_leaves`, and the spine builder that assembles decoded runs. Peeling and beneath change only a prefix, which leaves S valid. The field must cost nothing: every node pays for a field its body declares, leaves included, unless the field fits in padding the node already has. It does on illumos x86_64, where it was measured: `NodeInner` stays 216 bytes. A const assertion compares `NodeInner` with a twin type laid out without the count (`NodeInnerWithoutSpread`, `BodyWithoutSpread`, never built), so a platform where the count does not fit in padding fails the build, and the cost of the field becomes a decision rather than drift. No byte count is written: the compiler measures both layouts. A field added to `NodeInner` or `Body` must be added to the twins as well, or the assertion fails for that reason; its comment says so. Backend contract: `Node::grandchildren` and `ErasedNode::grandchildren` join len as required exact aggregates, with the reason a session needs them and what an undercount does; every implementor forwards it. Tests: - untyped: a direct recount at every node and every peeled view (`miscounted_grandchildren`, an explicit-stack traversal) over arbitrary trees and over both bulk builders' runs at every depth; the largest count, 65,536, at a full fan of full fans. - tree: replicas built by inserts and redactions, and their join. - streaming: both sides of reconciled pairs, in process and over a pipe, which builds nodes through assembly, the deletion filter, and the decoder's run assembly. - conformance: the wrapper recounts each listed node's grandchildren by exploding its children through the wrapped backend, tallying them as a u64 counter, and a backend that overcounts fails the check (`GRANDCHILDREN_SKEW`).
A ledger that charges before work allocates needs every built allocation's size, and node explosions are the one whose size only the backend knows: the in-memory backend copies a shared compressed node to peel a byte off its path, and the session cannot price that copy. `ErasedNode::explosion_bytes` now reports, before listing a node's children, the bytes the listing will allocate beyond the handles it yields. For the in-memory backend that is zero for a bare branch or an unshared node (peeled in place), and otherwise one node allocation (`NODE_ALLOCATION_BYTES`, its reference counts beside its storage) plus the copy's fan heap slots (`fan::heap_slots` of its length, at `FAN_ENTRY_BYTES` each). Another thread releasing a handle meanwhile can only turn a copy into an in-place peel, so the report never falls short. It joins the backend contract beside len and grandchildren; every implementor forwards it. `ops::children_of`, the one place a session explodes a node, reports the value through a new process hook (`holding::report_allocations`, beside the attribution hook) as work for `Holding::ExplodedNodes`. Without `test-internals` the report is a no-op. The session-budget suite, which arrives with the per-place heap measurement, holds the reports to what each side's exploded-node label allocates.
Mechanical: a listing entry was a (u8, Hash) tuple everywhere a question
or greeting lists children. The next commit gives each entry a third
part, the lister's grandchild count, which a tuple would carry as a
positional field read in dozens of places. Listed { radix, hash } names
the parts instead.
Every listing type becomes Vec<Listed>, and every site that built or
took apart a tuple builds or matches the struct; the window prices a
listing entry at size_of::<Listed>(), the same 25 bytes. Tests build
entries through a test-only listed(radix, hash) constructor. No
behavior, format, or size changes.
Format change (wire): every listing entry, in query frames and in the
greeting's root listing, becomes {radix: [hash, G - 1]}, and the
greeting gains a "grandchildren" field, the sender's grandchild count of
its root. The insta snapshots, the frame atlas, the corpus manifest, and
all 16 protocol_overhead cells are re-accepted here for this change: each
listing entry grows by its array head and its count's CBOR head, the
greeting by the root count, and nothing else moves.
BOOKMARK_FORMAT_VERSION is unaffected: a bookmark holds no listing.
Why: a ledger that reserves, before asking a question, the most memory
its reply can need must know that reply's size. A reply about a node
holds at most the asker's listed children plus the replier's grandchild
count G of it in reactions, and at most G listing entries. Only the
replier knows G, so it travels in the replier's own listing of the
child, one question earlier; the greeting carries the root's G for the
opening question.
What changes:
- message: `Listed` gains `grandchildren`, a `Grandchildren` newtype
holding G - 1 in a u16 (G in 1..=65,536). `Grandchildren::of(node)` is
max(1, G): a listed bare leaf, whose G is zero, counts one, which is
also the bound on a leaf request's reply (one supply). So every entry
has one wire shape and decodes without knowing its depth. `Greeting`
gains `grandchildren` (plain u64, 0 for an empty tree; the decoder
rejects more than 65,536). Its key sorts between version and
max_version_bytes.
- codec: `listing_entry_len` takes the count; `ListingBuilder` judges
the entry's array head, hash head, and count head (new structure
errors `Entry` and `Count`). The async reader still never reads past a
listing: it owes each entry's narrowest width until the count's first
byte arrives. `FULL_FAN_QUERY_FRAME_LEN` takes every count at its
widest, so `DEFAULT_TARGET_MESSAGE_SIZE`, defined as 256 such frames,
becomes 2,092,544 (from 1,830,400). The definition is unchanged.
- adapter: `Scope` keeps each listed child's count and a bound, the
replier's G of the node asked about. `derive` takes the bound from the
answered record's position, so the encoder's record of our question
about a child carries the peer's count of it. `Scope::reply_bound`
(`ReplyBound`) is: reactions min(256, listed + G), listing G; listing
0 for a leaf's parent; reactions min(256, G), listing 0 for an empty
question; one supply for a leaf. The decoder fails a reply at the
first reaction or listing entry over its bound with
`DecodeError::ExceedsBound` (boxed inside `ProtocolViolation`, so no
public type changes). The proxy threads the remote greeting's root
count to the opening record; the responder's replayed opening is never
decoded against, so its bound is MAX.
- window: a listing entry is priced at `size_of::<Listed>()` (28
bytes, from 25) and a record child at `size_of::<(u8,
Grandchildren)>()`, so `REFERENCE_SLOT_BYTES` is 68 (the conformance
suite's hand-computed 64-bit layout follows). The generated trade-off
table is regenerated.
- The grandchild count is now read, so its dead-code expectations go.
- reconciliation guide: the greeting's fields include the root's
grandchild count, and each entry of a disputed node's listing carries
its child's grandchild count, which bounds the reply a question about
that child can draw.
Faults: `GreetingRewrite` can understate or overstate the root count or
every root-listing count. Understatements fail at the deceived side's
decoder with `ExceedsBound` (the opening reply; a depth-1 reply);
overstatements complete with the unaltered join.
Measurements and pins that move with the format, each re-measured:
- dispute_wire: the reference fixture's mean is 145.362 B, so
`DISPUTE_OVERHEAD_BYTES` is 45 (from 43); the small-record cell sits
2 B below the estimate (from 1). The 75,000-message corpus measures
42.553 B, 2.45 B below the estimate, past the old 2 B tolerance:
listings, now heavier, weigh more in the dispute-heavy reference
fixture than in this supply-dominated one, so the tolerance is 3 B.
- parking: a maximally disputed reply's encoded-plus-decoded transient
is 3,796,480 B; its ceiling moves to 3,910,000 B (~3% headroom).
- The sizing guide's per-message figure is 45 + m.
- gossip_snapshot's listing counter reads the entry arrays.
Admitting work against a memory budget needs an account of what a session holds, so the account lands first: a ledger that charges and releases alongside the window, which still decides how much is in flight. The ledger (`streaming::ledger`). Each endpoint keeps one `Ledger`, which its walk creates and its proxy adopts (`ShareWindow::share_ledger` and `adopt_ledger`; `descend` shares it). A `Charge` is a number of bytes for one place (`Holding`), and it travels with the item it pays for: dropping the item drops the charge, which returns the bytes. So every charge is released exactly where its item is dropped. Prices count what an item keeps while it waits where the walk leaves it: its node in its channel (`queued::<T>`, the item padded to a word plus a link word, as futures-channel allocates per message) and its vectors' buffers at their capacities (`buffer`). `ReplyBound` moves here from the adapter as the one bound formula the decoder's check and the reservations share. What is charged, and where it returns: - Built items (`work/charges.rs`): an outgoing `Reply` carries its charge until its consumer is done (the proxy's encoder keeps it until the reply's last frame is out; a walk stage until it has finished the reply). A queued `Query` carries the charge for its node, its children, and the node copies listing them made (`explosion_bytes`). A `Resolution` carries its node and slots, and keeps the copies its answered node's explosion made until assembly. The proxy charges each record of a question it sends (`encode::Records`), carries the charge in `Parked`, and drops it at the take. - Reservations: each question carries the room its reply can need at the bound the peer's grandchild count sets: the reply parked whole, the records its take derives, and the parent resolution. The `Resolver` sizes the parent resolution's slots at the bound, and when it finishes the reply moves the question's charge and the reservation's parent-resolution part into that resolution (returned at assembly); the rest of the reservation returns then. Leaf requests carry theirs (`LeafRequest`). The opening question reserves from the responder's greeting count (`PeerSummary` keeps it); the greeting's root explosion is measured when the fan is built (`greeting_fan` returns its copies). - Fixed state (wire buffering, decode fans, assembly) is not charged. The resolver reports each finished reply's reactions and listing entries against its bound to a test hook (`observe_fits`); every ledger change goes to another (`observe_ledgers`). The ledger's unit tests hold its accounting to a model of the charges, and the descent oracle holds the replies it prices to what the walk builds. Each charge is a field of the item it pays for, so the queued items grow, and the window, which prices them by their size, moves: the generated trade-off table is regenerated. The default budget's window still covers the sizing guide's whole example population. Known gaps, for review: a request reply's charge prices the deletion filter's work by the top node's explosion only, so a filter that rebuilds deeper nodes is under-priced; and a ledger's account is not yet held to the heap it covers.
Previously a reply crossed every receive-side stage whole: the decoder read frames until the reply's end, parking held decoded replies, the take derived all of a reply's question records at once, and each walk stage began a reply only once all of it had arrived. A sender that paused partway through a reply, waiting for its earlier questions' answers, would deadlock against such a receiver. Now a reply is a run of chunks, the last marked (`Reply::last`): - The decoder delivers a reply's complete reactions as a chunk whenever the next frame has not arrived and one of them is a question, and the reply's remainder at its end; a supply completes when its group of consecutive supplies ends, at the next positional reaction or the reply's end, so a question never waits behind a supply. Its reader and assembler now talk over two edges, leaves one way and assembled supplies the other (`QueueKind::AssembledSupplies`, a fan of slots, which liveness needs). - Decoding and parking run in one task (`Ingress::decode_and_park`), and parking is unbounded (`channel::UNBOUNDED`): its unit is now a chunk, whose count the old in-flight count of replies no longer bounds, and every chunk lies within the room its question reserved. The parking overflow error, the parking capacity, and the tests of them go, and with them the test-only parking census, which recorded each queue's derived capacity: there is no capacity left to hold to a window, so the mismatched-window proptest checks only that both sides converge. - The take keeps the record a reply answers across its chunks and derives each chunk's records as it goes (`derive_questions` takes the positions already taken). The proxy trace records a later chunk's take as `TakenChunk`. - Every walk stage and the terminal absorb loop over a reply's chunks before finishing its scope. - The encoder carries a reply's state across chunks (`ReplyEncoder`): each chunk's frames are released with it, flagged to continue, and only the reply's last reaction carries the end, so a sender's last chunk must hold its last reaction. The reservation prices one parked chunk per question the reply can ask plus one, and a question record by the larger of its queue node and the allocation the decoder shares with the take. Tests: the descent model predicts every reply with every reply split at every reaction boundary where its consumer takes it, in process and over a one-byte pipe; the stress fixtures complete split the same way; a reply encoded in arbitrary chunks writes exactly its whole encoding's frames; a held stage leaves every invited reply parked; and the stall negative control is now a decoder whose supplies' edge cannot hold a group, beside the same session completing at a fan of supply slots.
Previously the walk built a whole answer before yielding it: it explored every disputed child, built every question's listing and query, yielded the reply, then published the answer's resolution and its queries. An answer's built items were therefore all of its questions' at once, and the encoder published a reply's question records only after its last frame. Now an answer is classified first (`answer::internal`, `answer::leaf_parent`): matches, supplies, and the resolution are complete, and each question is planned but unbuilt. The walk then yields the answer in chunks (`Chunks`, `Part`), each ending at one question, which it builds only when the chunk is reached (`answer::question`, `answer::leaf_question`), so a chunk's built items are its own question's: it explores a disputed child then. Each chunk's wire action comes first, then, after the first chunk alone, the answer's resolution, then the chunk's question (`yield_chunk!`). The reply's last chunk holds its last reaction, as the encoder requires. - The encoder publishes each question's record right after the frame carrying it, so the decoder holds the record before the peer can answer, even while the rest of the reply waits. The record queue is unbounded, so publishing never stops an encoder partway through a reply; the proxy no longer reads the window at all. - The walk's trace records a reply's later chunks (`Kind::Chunk`) and checks that each question follows the chunk carrying it and that later chunks follow the resolution. The in-flight count, which only parking's capacity needed, goes with its meter events and tests. The proxy's trace records each question's frame (`WireFrame`). - The transcript places a reply at its first chunk and extends it with the later ones. - Fault injection corrupts a whole first reply, joining its chunks. With the window still governing, the floor window's one-slot question queues now release an answer's questions one at a time, so the zero-budget catch-up corners take several times the hops they took, and the corners suite's hop bound is re-measured.
…r inside it
Previously the session window sized the walk's question and resolution
queues, and so decided how much was in flight, while the ledger only
accounted.
Now each endpoint's ledger admits the walk's work against that side's own
budget (`ledger::admission`). The walk builds in units: each answer's own
items (its resolution, the copies exploding its scope makes, and the
reactions that ask nothing) are one unit, and each question it asks (its
query, chunk, record, and the reservation for the peer's reply) is
another. A unit waits for admission before it is built and is granted a
charge of the most it can hold; every item is built out of its unit's
grant (`Charge::take`), and what is left returns once the unit is built.
The rule lives in `admission::decide`:
1. A unit takes its depth's floor slot for its kind if the slot is free;
the slot stays taken until every byte built out of its grant is
released, and those bytes are the ledger's floor part, which the
budget does not count.
2. Otherwise it is admitted against the budget if it fits beside what the
ledger holds outside its floor slots and no deeper unit waits.
3. Otherwise it waits.
Each depth has two slots, one for answers and one for questions, both
indexed by the answering scope's depth: an answer's own items, released
at its scope's assembly, which waits on its later questions, cannot share
a slot with them, and the module docs give the induction for the rest.
The floor counts inside the budget. A session that may hold M gives its
ledger the budget M - A, where A is the floor allowance, the most the
floor slots can hold at once, derived from the walk's types:
- charges::slot_bound bounds each floor slot by the largest grant the
walk can pass a unit of its kind at its depth, from the same bounds the
walk admits with: the opening question, unled, at the root; every other
question at a fan of children about a node with the type maximum of
grandchildren, behind two fans of reactions; a leaf request at the
leaves; an answer exploring or supplying a fan, or one leaf at the
leaves; every explosion at the backend's bound.
- charges::floor_allowance sums one side's slots: a side asks only at
depths of one parity and answers only at the other, the initiator
asking at even depths from the opening question down. The ledger is
built before the handshake settles the side, so the allowance is the
larger side's: 35,576,192 B (about 34 MiB) on the in-memory backend,
the replies' listings, a fan of fans each, nearly all of it.
- WindowConfig::admission_budget gives the ledger a configured budget
less the allowance, saturating at zero, so a budget at or below the
allowance runs on the floor slots alone and holds at most the
allowance. A fixed test window admits against zero at the floor and an
unlimited budget otherwise.
- ErasedNode::MAX_EXPLOSION_BYTES bounds what any explosion copies, so
the allowance covers the copies the units charge: on the in-memory
backend, one node allocation beside a full fan's heap slots, the copy
of a shared branch with every child.
So each side holds at most its budget: what the ledger holds outside its
floor slots is at most M - A, and each slot holds at most one grant of
its kind and depth.
A freed floor slot carries what its kind and depth hold against the
budget. A unit whose slot is taken is admitted against the budget, and the
slot can free moments later, even within the same instant, while that
unit still holds its grant; its bytes would then hold budget beside an
idle slot for the rest of the unit's life, for no work of their own. A
budget one byte larger than such a unit's grant turns a wait for the slot
into this substitution, so the larger budget runs slower: in a disjoint
pair of 2,000,000 messages a side on a LAN, at ledger budgets of 18,647
and 18,648 B, the initiator's depth-1 answers took the budget instead of
their slot at 23 subtree boundaries, each affected subtree nearly
doubled, and the session ran 11,435 one-way delays longer (52,918 on the
latency-only link). These figures come from the budget-curves
simulator, which later commits add, run against this rule without
carrying. So a slot that frees while no unit of its kind and
depth waits for it carries the bytes its kind and depth hold against the
budget, up to the largest grant admitted to one of them: the budget stops
counting them, the floor part counts them, and the slot stays taken until
as many bytes of its kind and depth have returned. A charge names its
unit's kind and depth, not the unit, so a slot carries a count of bytes,
and every charge of its kind and depth admitted against the budget
returns its bytes to that count first. A unit waiting for the slot still
takes it first. The module docs ("Carrying") state where each ledger
invariant holds: carrying moves no bytes between charges, reads the
schedule and never the budget, and frees each slot no later than the
units it carried for release their charges, so every unit is still
admitted.
Every admission decision reads lock-guarded state. Computing the budget's
count as held - floor - carried, from three counters a floor charge's
drop changes outside the admission queue's lock, lets a reader on weakly
ordered hardware see a new total beside an old floor part: in debug
builds the subtraction panics, a panic triggered by scheduling, and in
release builds it wraps and admission refuses. So the bytes the budget
counts, whether each slot is taken, what each kind and depth holds
against the budget, and what each slot carries live in the queue's locked
state (`admission::Slots`), which `Account::add`, `Account::sub` and
`Account::carry` take, so the compiler ties every change to the lock. A
charge is built under the lock; an item's shortfall joins under it
(`admission::shortfall`); and every drop returns its bytes, leaves its
slot, and reconsiders the waiting units in one critical section
(`admission::release`), as does a charge folded into another
(`admission::fold`). A floor charge holds its slot through a shared
pointer whose count is the slot's sharers, and the slot frees only as the
last one is taken under the lock. The ledger's total, floor part and
carried sum remain atomics, read only by tests and the observation hook.
- The ledger's cumulative counters, the bytes items needed beyond their
grants and the units that waited, are u64 (`crate::counter`).
- The public `Peer` budget docs keep their window-era text; the commit
that removes the window restates them.
- The walk's question and resolution queues are unbounded, since the
ledger charges what they hold; the window sizes no queue, so the walk
and the proxy drop it, and the public `SessionStats::window_stalls`
reads zero. The window still resolves for the stats it reports.
- The proxy keeps no ledger: each question's chunk carries the record
charge its walk built (`Reply::record`), which the encoder publishes
the record with. The participants no longer share a ledger.
- Every stage releases a chunk's charges before it waits for the next
chunk, since the peer's admission may wait on them. Fault injection
corrupts a reply in place rather than buffering its chunks, which would
hold a producer behind its consumer.
- Each part's leading reactions are allocated exactly, so a chunk holds
no more than its unit's bound prices.
- A charge names its floor slot and its unit's kind and depth, and so
grows from 24 to 32 bytes. The window prices queued queries and
resolutions at their sizes, so its generated trade-off table
(`window/tradeoff.md`) moves; it is regenerated with
`just window-tradeoff`.
Tests: the admission rule, its slots, deeper-first budget, withdrawal,
carrying and a waiting unit's priority over it, a shortfall carried and
returned in full, and a property holding the rule, the bound, the
budget's count against the live charges, and each carry against what its
kind and depth hold and the largest grant, through any sequence of
admissions, splits, folds, drops and withdrawals; a cross-thread test
passing one slot between eight threads at a zero budget checks that a
slot frees only in the critical section returning its last bytes; the
ledger proptest checks the budget's count and the shortfall; a test holds
the floor allowance to its documented figure; a closed-world grid of wire
sessions across sizes, overlaps, pipes of one byte through whole frames,
and zero, tiny, mismatched and unlimited budget pairs, with forks placed
at many depths; each of its budgets is what a side's ledger admits
against, so each session's memory budget is it plus the floor
allowance. The capacity laws are re-measured under a zero budget: a
lone scope stalls past a fan of its return capacity plus one, and the
full-fan witness needs a fan less one. The window's session test keeps
its population check and drops its check of queue widths, which the
window no longer sets.
Pins and generated artifacts that move:
- tests/window/corners.rs: CATCH_UP_HOPS, 58 -> 160. At a zero budget a
catch-up's requests at one depth each take that depth's one question
floor slot, which frees only once the reply it invites is processed, so
they cost about a round trip each where the window's one-slot queue let
them leave back to back; the reverse direction measures 140. The bound
still does not grow with the supplied messages.
- window/tradeoff.md: regenerated, since a charge's 32 bytes enter the
window's prices for queued queries and resolutions.
- The window's knee suite (tests/window/knee.rs) goes: its claims are
that the window's derived queue capacities set where a session stops
pipelining, and the window sizes no queue now. The window's session
test likewise drops its queue-width check.
Reading order: the `admission` module docs (the rule, the floor
allowance, carrying, the lock, and the deadlock argument), `decide`, the
slots and `release`, then `ledger.rs`, then the walk's unit boundaries in
`materialized/work/levels.rs` and `answer.rs` with their bounds in
`charges.rs`, then the grid (`streaming/tests/grid.rs`). The window's
files can be skipped.
Two claims about a session's speed need checking on the link model: that a budget large enough for everything a session admits costs it nothing, and that unsent bytes in a sending kernel's buffer do not slow it. Both are stated in the bytes the ledgers admit against, since admission, not a window, decides how much is in flight. The fixture reads each session's ledgers through the ledger hook, per driving thread: the most either side holds outside its floor slots, and the units that waited for admission. Its pairs can share messages, and `admitting(bytes)` configures the memory budget whose ledgers admit against `bytes`: `bytes` beside the floor allowance every budget sets aside (`testing::floor_allowance`). - The no-slowdown suite (`tests/throughput/no_slowdown.rs`) asserts an identity. For pairs of 10³ and 10⁴ messages a side, disjoint and 90% shared, on a LAN (125 kB) and a long-haul link (12.5 MB), the oracle runs with an unlimited budget and reads Λ, the most a side's ledger holds outside its floor slots; it must have made no unit wait. The same session with both ledgers admitting against Λ must then carry the oracle's every segment at the oracle's every instant: with B ≥ Λ no admission against the budget is refused, and the floor slots' use does not depend on the budget. - The send-buffer suite (`tests/throughput/send_buffer.rs`) checks that unsent kernel bytes do not slow a session, at a budget of Λ, where no unit waits, and at half of it, where units wait for admission, which the test checks; and on a catch-up whose requests queue behind a side's own supplies. A side's requests and its own supplies share one ordered stream, so an analysis predicted that a bandwidth-delay product of unsent bytes ahead of the requests would stretch a fully divergent session by half its transfer again; the suite checks that predicted excess is far beyond the model's resolution, so its verdict cannot rest on that resolution, and that the session takes no more than its duration without unsent bytes plus that resolution.
Previously a peer, a bootstrap builder, and the walk's builder held a
window choice (`WindowConfig`): a byte budget, or, in tests, a fixed
table of per-height capacities, the one-slot floor among them. Since
admission, only the budget a choice implies acts on a session (its
ledger admits against the budget less the floor allowance, a floor table
against zero, any other table against no limit), and a fixed table's
widths size nothing.
Now each holds its budget in bytes. A peer and a bootstrap builder hold
their memory budget (`sync_memory_budget`), and the walk's builder takes
it the same way (`Handshaking::sync_memory_budget`), giving its ledger
the budget less the floor allowance. A driver that states the budget its
ledger admits against sets it directly (`Handshaking::budget`), as the
streaming tests and the closed-world grid do, so the grid's budgets need
no allowance added. The window statistic is priced from the memory
budget, or from the ledger's budget where only that is given.
`WindowConfig`, the test-only `Peer::sync_window_floor`, and
`Window::uniform` go.
The test surface follows, preserving what each session ran:
- The one-slot floor becomes a budget of zero (`sync_memory_budget(0)`),
and a wider fixed table an unlimited budget. Suites that drew a width
(one, two, or eight) draw a budget of zero or an unlimited one, which
is what those widths admitted against.
- The test kit's sweep dimension is a budget (`common::budget`:
`BudgetChoice::{Zero, Bytes, Default}`, `BudgetAssignment`,
`bootstrap_fork_with_budget`). Its explicit budgets are drawn beside
the floor allowance, so each admits some work against the budget; the
window sweep's population test reads them past the allowance.
- The join-oracle property's unlimited arm no longer checks the width
the session reported, a statement about the window, and the window's
unit tests of `WindowConfig` itself go with it.
- Prose that described a test's configuration as a window now states its
budget. Two pieces of prose described parking as capped at one slot or
as able to overflow, which it has not been since it became unbounded;
they now say it is unbounded.
Previously each session still solved a window from the greetings' set sizes and its memory budget, through the statistics of a hash-distributed tree (quantiles of occupancy and child counts), handed it from walk to proxy, and reported its widest capacity (`SessionStats::window_granted`). Since admission, the window sized no queue and limited nothing: the ledger admits each unit of work against the budget less the floor allowance. The budget docs and the sizing guide still described the window: a sizing target estimated from the two sets, a trade-off table, and a slowdown model over the window's width. Now the budget is only the admission budget, and its docs present it as the memory a session spends on pipelining. Removed: - the window (`Window`), its solve and slot prices, the replica sizes it read (`ReplicaSize`), and the hand-off of a resolved window from walk to proxy (`ShareWindow`); - `SessionStats::window_granted` and `window_stalls` (public), and the recorder's window counter; - the trade-off table, its example, the `window-tradeoff` recipe, the window's own tests and seed file, and the wall-clock bench of the window suites; - the window suites whose claims were the window's: the census (the budget-to-capacity solve) and the sweep (that budgets widen the granted window); the floor-window statistic test; - the test-only `testing` helpers that served them (`window_tradeoff_table`, `window_capacities`, `max_version_bytes`, `max_bound_bytes`); - the backend conformance suite's slot-padding obligation: the window priced every slot at constants derived for the in-memory handle and left a wider handle's padding to the backend's node price, while the ledger charges each item at its size at the backend's own types. Moved: `FAN` to the streaming module, the default budget to the ledger, the decode fans' bound and its record count beside the decoder (the bound now computed from the decoded leaf's type), and the dispute-overhead figures into `testing`, beside the helpers that report them. Kept, compiled only for tests: `Backend::node_bytes`. No session reads the price; the backend conformance suite still checks the contract. Docs: - `Peer::sync_memory_budget` and the sizing guide present the budget as the memory spent on pipelining: a budget of zero asks one question at a time per level, by declaration; more keeps more in flight until nothing waits; the budget counts room set aside for answers, so nearly identical peers can slow before memory reaches it. - They state what the budget covers: the work a session has built (its questions, its unsent answers, its question records, and its node copies), the room it sets aside for the answers it invited, and the floor allowance, room for one question and one answer per level that a session can always hold. The floor counts inside the budget: a session admits the rest of its work against the budget less the allowance, about 34 MiB on the in-memory replica on 64-bit targets, so any budget at or below it behaves as zero. - The guide's table and slowdown model go; no doc claims what a given budget achieves on a given link. - `target_message_size` no longer says the decode fans' leaves are charged against the budget: the decode fans hold a fixed number of leaves per level, whatever the budget. - The adapter's module doc names the decoder's memory account as the fan of decoded leaves alone: no backend prices them, and no supply-decode charge remains. - `reconciliation.rs`, the conformance suite, and the module and test docs that described the window now describe admission. Tests: - `tests/window/corners.rs` becomes `tests/budget/corners.rs`, and its budget-zero tests assert that each session completes and converges, printing the hops it took rather than bounding them: a budget of zero means one question at a time, by declaration, so its speed is the declared behavior, not a property to bound. The growth test runs on the default budget and keeps its claims that the racing session completes, a racing commit lands after the greeting, and the next session converges. - `tests/window/pipelining.rs` becomes `tests/budget/pipelining.rs`: it holds the default budget under its hop ceiling and prints, without asserting, the same session at budget zero. - The backend conformance census checks the peak increase alone; its window-widening test goes. No wire format moves: the greeting carries no budget, and no capture changes.
…y place The budget claims to cap the memory one session uses beyond the replica, and the ledger claims to cover the session's transient heap, less its fixed state and the scratch its stages fill, at every moment. Nothing measured either claim against the heap. The new `session_budget` suite (`tests/session_budget.rs`) drives two fully divergent replicas through one session over a held link (`tests/session_budget/link.rs`). The link releases bytes only when both peers stall, so each side fills whatever its admission lets it have outstanding before any reply arrives, and each release hands the reader everything its peer produced meanwhile. The driver polls each peer on the calling thread under its own allocation label, and the attribution hook adds the place each marked region serves, so a trace labels every allocation with its peer and place. The link's buffers and the driver's state are allocated outside the trace, so every block the trace records is some peer's. After the peers drop, with snapshots holding their replicas, each block the session allocated is classified: a block freed during the session was transient, and a survivor is replica content or state the peer kept. Every label has a bound, and each side's whole transient peak must stay within their sum: - the places the ledger charges, together, within the ledger's bound (the budget it admits against beside its floor slots' largest grants) plus each place's scratch; - wire buffering within the frames the session holds per stream, each at most the largest comparison frame, which the fixture's run target equals; - assembly within `testing::assembly_allowance`, and the decode fans within a fan of leaves per stream; - the unattributed blocks, the session's setup, within a reference session's whole transient peak. The suite holds every session to the ledger's identities, at every allocation event: - Coverage, place by place: each charged place's metered transient bytes may exceed the ledger's charges for that place by at most the place's scratch, which `materialized::scratch` computes from the types it covers, per logical stream (each channel's lingering stub node, an answer's plan, a chunk's reactions before its question is admitted, a question's listing before its chunk is charged, a decoder's ready reactions, a take's derived scopes, and the record an encoder still holds). The exploded nodes' place also allows the greeting's copy of the root, made before any unit is admitted (`testing::ledger_scratch`). Checking each place alone keeps room set aside in one place from hiding a charge missing in another; the total stays checked too. - Fit: every reply a stage finishes fits the room its question reserved. - Zero: each ledger holds nothing once its session ends. - The bound: each ledger holds at most its budget outside its floor slots, and at most its peer's memory budget in all, the budget beside the floor allowance (`testing::floor_allowance`). Each side must have counted, finished some reply, copied some node, and made some unit wait for admission, so no check passes vacuously; each side's survivors must number at least the messages it gained, and the explosion bytes its backend reports must sum to its exploded-node label exactly. So that each place's charges match what the place holds: - A reservation is charged in three parts (`Reserved`): the parked chunks, the records the take derives, and the parent resolution's slots, each to the place its items take; a leaf request's queue slot is charged to the questions. - A question's node copies are charged to the exploded nodes' place, and every resolution keeps them separately (`Resolution::copies`), as does a supply chunk for the copies its supplies made (`Reply::copies`). - A ledger change reports every place's holding (`LedgerChange::places`), a move between places included. A slow storage backend can stall walk stages or assembly while bytes keep arriving, which no link schedule produces alone. With `test-internals`, every session task runs through `gated` (`streaming/tasks.rs`), which asks a hook installed with `testing::gate_tasks` whether the task's `TaskRole` may run on each poll; a role names a walk stage or assembler by height, an encoder, or a decoder by height. Without the feature, or without a hook, `gated` is the future it wraps. The held link's schedule chooses, at each stall, which kinds of each side's tasks to hold back until the next (`Held`), and the driver's gate holds them as a waiting backend would. Cases run at 8 MiB, near the floor at 10⁵ messages a side; at a wide 32 MiB budget; with each side's walk stages, and then its assembly, held back at 32 MiB, each required to raise its place's peak above the unheld session's; with a peer joining from empty; and as a property over release schedules and held-back task kinds. The property builds its own runner, so `PROPTEST_CASES` cannot override its fixed case count. The committed figures are measured at this commit, under the admission rule that admits against the budget less the floor allowance and carries a finished floor slot's grant: - ledger peaks at 8 MiB: 10,219,296 and 8,593,248 B; at 32 MiB: 35,397,992 and 33,663,964 B (each the budget filled with admitted work, beside what the floor slots hold and carry); - ledger peaks when a peer joins from empty: 13,144 and 12,616 B. No heap peak is committed: heap peaks differ between macOS and illumos, while the ledger peaks agree, so each held-back fixture first runs the same session with nothing held back, in the same test, and requires its own peak to exceed that run's. The added charges grow each unit's grant, and with it the floor allowance, from 35,576,192 to 35,711,552 B, still the documented "about 34 MiB". Supporting changes: the meter's analysis is optimized in the dev profile, since classifying tens of millions of events dominates the suite (`Cargo.toml`); `bootstrap_fork_then` joins a fork and configures it with any closure (`rumors-testkit`'s `common/wire.rs`); `testing` gains the suite's bounds (`SESSION_STREAMS`, `COMPARISON_FRAME_BYTES`, `assembly_allowance`, `ledger_scratch`) and the task gate; the ledger's module docs name the suite as what holds the ledger to its coverage; and a comment in `src/conformance.rs` names the backend suite's Scope section.
Trait change: `Backend::node_bytes` is removed from the crate-internal backend contract. Previously a backend priced what each held node value keeps resident (`node_bytes`, by child count and version-bound width), and the window priced its queues with it. Now the ledger charges each item by its in-memory size and each node copy by the backend's report of its explosion (`ErasedNode::explosion_bytes`), and nothing reads the price: it is compiled only for the tests that check it, so it goes. Removed with it: - the conformance suite's price checks: the pointwise comparisons of measured residency against the price at leaf construction, at each parent, at the explosion, in the leaf walk and in bulk assembly, and the monotonicity sweep over child counts and version bounds with its grid; - the tests of those checks (an underpriced header; over-holding walk, children, and assembly rows; an underpriced leaf; a dip at one fan; and the bound-monotonicity property with its point- and step-dip controls), the reference backend's knobs that only they set, and the seed file of the monotonicity property; - the in-memory and failing backends' implementations. The conformance suite keeps its residency census, its re-tag residency checks, every operation check, and the reconciliation checks; its docs and the backend docs no longer mention prices.
0.5 changes only num-bigint's random-number support: the rand feature splits into rand_0_9 and rand_0_10 (with rand_core-only variants), RandBigInt becomes one extension trait per rand release (BigRng010 for rand 0.10), and the gen_* methods are deprecated in favour of random_*. Its arithmetic is the same source as 0.4.8, the release the lockfile resolved, so before's and suanpan's integer code compiles and runs as before. suanpan's num-bigint feature exposes num-bigint's integers in its public API, so that feature now speaks 0.5. num-bigint 0.5 has no rand 0.8 support, so the fuelscape's sampler moves from the rand 0.8 and rand_chacha 0.3 aliases to the workspace's rand 0.10 and rand_chacha 0.10, drawing bounded integers through BigRng010 and ranges through RngExt::random_range. The per-cell seed expansion and the ChaCha20 key stream are unchanged, but rand calls this release value-breaking: the same cell seed now draws different samples. No committed result depends on the old draws: the uniformity pins hold for the new stream, and the committed dump is replayed and recompacted, never resampled. The fuzz-fit harness still uses the aliases, for its calibration corpus. cbor-diag, the wire-capture renderer behind test-internals, still requires num-bigint 0.4, so the root lockfile keeps 0.4.8 beside 0.5.1. The detached fuzz workspace builds before by path, so its lockfile moves to 0.5.1.
idna_adapter's version selects the Unicode backend of idna, which url uses, and url reaches the tree only through cbor-diag, the wire-capture renderer behind test-internals. 1.1 selects the unicode-rs crates; 1.2 selects ICU4X, idna's default backend. The lockfile held 1.1 to keep ICU4X's derive macros from adding a second syn major to the shipped closure, but both syn majors now ship through other crates (async-stream on syn 2, futures on syn 3), so the hold keeps nothing out. With 1.2 the lockfile gains the ICU4X crates and drops unicode-bidi, unicode-normalization and unicode-joining-type; no crate in the shipped closure is duplicated by the change. The lockfile is updated with `cargo update -p idna_adapter --precise 1.2.2`.
pollster 1.0 replaces block_on's own wake signal, a mutex and condition variable recording whether a wakeup arrived, with the thread's park token alone. A future that parks its own thread while it is polled then consumes any wakeup that arrives meanwhile, and block_on parks with nothing left to wake it. The peer suite's race between redaction and publication does exactly that: its hook runs inside a gossip session under block_on and waits on std::thread::scope, which parks the thread until the scope's threads finish. Under 1.0 that test hung in 43 of 96 runs on ox-east-1, its one remaining thread parked in block_on; under 0.4 all 96 passed. The workspace entry now says so, beside the other release this tree holds back for a stated reason.
A replica's observers end once every sender of its watch channel is gone, which try_into_peer's docs promise for a generation whose handles all drop: "observers drain the final state and stop". Every gossip session builds the guard that returns an unsent bootstrap fork, and that guard holds its own clone of the sender, whether or not the session serves a joining peer. A session future leaked after building the guard never drops it, so the replica's observers wait forever. The new proptest leaks a session parked in its checkpoint's bookmark store, drops the only handle, and drains an observer, which must yield both messages and then end. It fails until the next commit.
Previously the guard that returns an unsent bootstrap fork held its own clone of the replica's watch sender. Every gossip session builds that guard, so a session future leaked instead of dropped (with mem::forget, say) kept a sender alive forever, and the replica's observers never ended after every handle was gone, contrary to try_into_peer's docs. Now the guard borrows the sender from the session's peer. Nothing else changes: the guard lives inside the session future, which already borrows the peer for its whole life, and its destructor rejoins an abandoned fork through the borrow exactly as before. A leaked future holds no sender, so observers end once the last handle drops. The previous commit's test passes.
…e it The large-payload test sends two byte-vector payloads: one past serde's 1 MiB preallocation cap and, by its docs, within the default target's run budget, so that it travels in a run of its own beside the wire pools; and one past the largest budgeted message, which holds its excess beside the pools as it crosses. Nothing checked which side of the largest budgeted message each payload's message lies on, and the first no longer lies within it: since the default message-size target became 256 KiB, the largest budgeted message is 524,332 B, and the first payload's message, 1,114,117 B encoded, exceeds it by 590,809 B. Both payloads now take the lone-record path, and no session in the suite decodes a payload past serde's cap within the run budget. Each payload now states whether its message lies past the largest budgeted message, and the test checks that before running its session. The check fails here, on the first payload; the next commit moves that payload to a target whose run budget holds it.
… holds it The large-payload test's first payload, 1 MiB + 64 KiB, exists to decode past serde's preallocation cap while its message stays within the run budget, so that it crosses in a run of its own beside the wire pools. At the default target of 256 KiB no payload past that cap can stay within the run budget, whose largest budgeted message is about 512 KiB. Its peers now run at a 1 MiB target, whose largest budgeted message, about twice the target, holds it, and the preceding commit's check passes. The second payload stays at the default target, a mebibyte past its largest budgeted message. Each payload's wire bound and its lone record's excess are computed at its own target: - `testing` gains `budgeted_message_bytes`, the largest budgeted message at a given target, beside `wire_allowance`; - `payload_fixture` takes the target its peers run at; - the wire allowance, the excess and the liveness check that a payload past the run budget held more than one budgeted message all read the payload's target.
… per-message figures The guide said that on a LAN a session short of budget loses at most a few percent, that at a million messages on a 12.5 MB link a session takes up to about three times as long until its budget nears the point where nothing waits, and that replicas that differ throughout use about 400 bytes of budget per message before nothing waits (200 to 300 sharing 90%, 50 to 120 sharing 99%). Measured with the test kit's session runner on the simulated link, at the fixed network seed, two replicas of a million 100-byte messages that differ throughout took 2,779,715,616 byte-times at the smallest budget on the 12.5 MB link, 14.5 times the 192,059,458 of an unlimited budget, and on the LAN link 142,107,212 against 142,101,132. The slowdown figure is restated from those measurements, and the LAN's as the measured equality. The per-message figures were measured under an earlier admission rule, which the per-kind-and-depth rule replaced, and they are not re-measured here, so the guide keeps only that differing replicas use a few hundred bytes a message and mostly shared ones less, and keeps its advice to measure the workload.
…ly until reunion A session that serves a joining peer reserves a fork of its party for the joiner, and bookmark reclamation pauses while any reservation is outstanding, because an identity recorded before the reservation still contains the fork. The reservation drops when the session's fork guard does. A session future leaked instead of dropped (with mem::forget, say) never drops it, so reclamation stays paused for the replica's lifetime, even after the peer is reclaimed from its handles. The new proptest leaks a bootstrap service at a drawn point: before the fork is reserved, while the checkpoint's store waits, during reconciliation, or while the donation's store waits, so the fork is never sent. It then reclaims the peer through a drawn number of handles and reuniters, runs a session, and checks with the test kit's party oracle that the live parties partition the whole identity space. That holds only if the session reclaimed the leaked fork's region, which the bookmark still records. Leaks and reunions repeat, and a joiner served at the end must receive a fork without overlap. Supporting changes: - StallingBookmark can stall the nth store from now, not only the next, and can be disarmed when a leak lands before the armed store. - The leak tests share a helper for reclaiming the peer. - The test kit's party invariants are also available over parties already read, for a fleet that mixes bookmark types. The test fails until the next commit, on the partition check: the leaked region is missing from the union. The failing input replays from the seed already committed for this file, so no seed file changes.
…e peer Previously a session future leaked while serving a joining peer kept its fork reservation forever, because the fork guard that drops it never ran. Bookmark checkpoints reclaim nothing while a reservation is outstanding, so reclamation stayed paused until the process restarted, and the reclaimed peer could not repair it. Now reunion forgets every outstanding reservation, beside releasing an abandoned bookmark lock, through Reservations::release_abandoned, which replaces the count with a fresh one under the replica lock. Why this cannot hand a party out twice: when try_into_peer reclaims the peer, no session future of the generation can run or be dropped again, the premise the bookmark-lock release already rests on. So an outstanding reservation belongs to a leaked future. A session releases its reservation before it writes the first byte of the fork, so that fork was never sent, and the future can never send it or rejoin it. No live peer owns its region, which is the reclaim rule's first condition, and a later checkpoint that reclaims it from an identity still recording it gives it to the live party alone. The argument is stated at release_abandoned, at the reunion site, at the point where a session releases its reservation, and in the record's module docs beside the reclaim rule. try_into_peer's "Leaked sessions" section says that reclaiming the peer resumes reclamation. No public API changes. The previous commit's test passes.
…e's speaker by From The marks that attribute a session's allocations to the place holding them become methods on the place: `Holding::create` runs a closure as that place's work, and `Holding::creating` and `Holding::creating_stream` poll a future or a stream as that place's work, every poll, in place of the free functions `held_by`, `holding` and `holding_stream`. A call site now reads from the place to the work it marks, as `TaskRole::paced` does for the task it paces. `From<Role> for Speaker` names an elected role by the codec's speaker, the inverse of `Speaker::role`, in place of two private `speaker` functions that each wrote the match out.
The maintainer docs used "take" as a noun, never defined, for the moment a walk stage takes a parked reply chunk out of parking: the proxy then derives a question record for each question the chunk asks and publishes those records to the encoder's queue. Each such use now names the event plainly, as the stage taking the reply from parking, or as the records derived from a reply's questions, and the proxy is named wherever it is the actor. The bounded channel's "a take", meaning a receive, now says so, and the progress trace's panic message speaks of a taken answer, keeping the text its should_panic tests expect. Comments, doc comments and assertion messages only; no identifier changes, and no "record"/"scope" wording changes beyond what a rewritten sentence needed. Rewrapped paragraphs keep the 76-column width their neighbours use.
Previously, a bootstrap that received its whole identity and then failed
while the two sides confirmed completion returned Joined::Unconfirmed: a
peer standing exactly as a joined one does, but with its selected bookmark
left unattached and the connection's halves gone. What a developer acts on
in that case is three separate facts (the peer, whether the connection can
carry another session, and an error to log), and whether the provider
confirmed is not one of them.
Now Connection<R, W, B> carries the connection's fate as its own value:
Reusable { read, write } when the session completed, resting at the
session boundary, or Lost { error } when it failed after this side
joined. Joined::Joined and Joined::Unbookmarked carry a connection beside
their peer, and Unconfirmed is gone. Bookmark attachment runs whenever
there is a peer, so a lost connection after the identity arrived still
yields an attached peer, or Unbookmarked if attachment fails. Bailed and
Failed are unchanged: neither has a peer, so neither has a fate to report
apart from its variant.
Connection's B parametrizes its error, Error<B>, as Error's own B does;
retirement's errors carry the peer's bookmark type, bootstrap's are
Error<NoBookmark>. Lost uses a named field to match the error fields of
Joined and Retire.
Internally, the private Bootstrapped enum existed only to carry the old
split; bootstrap_inner now returns the received peer, if any, beside the
session's result, and join builds the outcome from that pair.
Docs: Joined, Bootstrap::join and Bootstrap::bookmark, Phase::Completion,
the error module, and the crate docs' "What a session promises" state the
new shape; README regenerated.
Tests: every match on the old success arm, Joined::Joined or
Joined::Unbookmarked, now also requires Connection::Reusable, since the
old arm meant the session completed; failure-tolerant sites accept any
peer, as they accepted Joined or Unconfirmed before. The cancelled-bootstrap
proptest's floor (some join keeps a party its provider never confirmed)
counts joins that return a peer beside a lost connection. The deadline
test selects a bookmark and also asserts the peer comes back attached with
a persisted record. Doc examples keep matching Joined::Joined { peer, .. }:
they want the peer whatever the connection's fate.
Previously, Retire split the departed peer's outcome on whether the
absorber confirmed: Retired { read, write } when it did, and Uncertain
{ error } when the session failed after the retiree began sending its
identity. Both mean the same thing for the identity: the peer was
consumed and cannot gossip or retry. What differs is whether the
connection can carry another session and an error to log, the same split
Joined had.
Now Retired { connection } carries a Connection<R, W, B>: Reusable when
the absorber confirmed completion, Lost { error } when the session failed
after the handoff began, including while awaiting Phase::Completion, in
which case the absorber may or may not have committed. Uncertain is gone.
Declined { peer, read, write } and Recovered { peer, error } are
unchanged: both return a peer that still holds its identity, and they
parallel Joined's Bailed and Failed, which keep the same form.
retire_inner maps every session that set the retire intent to Retired
and settles the connection with Connection::from_session.
Docs: Retire, Peer::retire and its lifecycle example, Bookmark's
recovery limits, and the crate docs' "What a session promises" state the
new shape. The lifecycle example logs a lost connection's error; with no
error path left, its Result scaffolding goes; README regenerated.
Tests: as for Joined, an old Retired match becomes Retired with
Connection::Reusable and an old Uncertain match becomes Retired with
Connection::Lost, so every claim keeps its strength. The severed-session
tests are renamed for what they check, that the severed retiree is
consumed; the cancelled-retirement proptest's private outcome enum names
its variants Confirmed and Lost. The fleet simulation and the causality
harness, which treated both outcomes alike, now match one arm.
Connection::into_halves returns a reusable connection's halves, or the error that lost the connection, so a caller who wants the connection for another session can pass a lost connection's error on with ?.
Comments, rustdoc, tool headers and the root AGENTS.md named the owner, their rulings, agents' worktrees, or called an artifact the one "of record". Each such sentence now states the thing itself: the budget criterion is the criterion a duration's movement must meet; a snapshot is re-accepted only for a deliberate format change named in the commit; a board cell is resolved by a fix or a model declared at the cell; the toolchain, the workspace dependency table and the cargo-rdme version are described by what they set; the board's gate verdict and calibration profile are named as such. Comments and documentation only; no behavior changes.
Comments, rustdoc, assertion messages and AGENTS.md in `before`, its instrument crates and `before-fuelscape` called artifacts the ones "of record", or cited an owner's ruling or review. Each now says which artifact is meant: the board's calibration architectures are the currencies' *required architectures*, defined at `HEAP_ARCHS`; the board verdict the gate reads is the acceptance verdict; build.rs writes each fuelscape island's contract text; the verdict matrix's reference verdict, the derived fuzz seed set, the calibration corpus and a family's registry row and name are named as such. Surface exceptions carry a reason, and a committed ceiling replaces a ratified one. Identifiers are unchanged, so `archs_of_record`, `text_of_record`, `texts_of_record`, the `of_record` locals and three test names still carry the phrase. Comments, documentation and assertion messages only; no behavior changes.
…ecided them The Lean docstrings and comments, MODEL.md, the READMEs, the lakefile and the exposition cited Finch's rulings, "decision-for-Finch" items, the adjudication, the cross-examination, the review panel and its synthesis, and called statements, strategies, grains, harnesses and models the ones "of record". Each such sentence now states the thing itself: the suite's claimed statements and the charter grain they bind; T5's oracle and its fallback form; the observation alphabet, in which a machine may peek at its slot and consumption receipts are excluded; the `Mux.apply` harness; the byte caveat stated in Mux/Basic.lean's module doc. The exposition states the conjecture instead of attributing it. The narrative and process documents (doc/narrative.typ, PLAN.md, PROGRESS.md) are untouched. Comments and documentation only; no definition or proof changes.
…s shape Its shrink note lacks fork_parents: the case it was written for no longer exists under the current strategy, so its hash replays nothing that checks the failure it recorded. The comment explaining why it stayed goes with it.
…eed set The docs call them the currency's required architectures, the islands' contract texts and the derived seed set; the identifiers now say the same (`required_archs`, `contract_text`, `contract_texts`, and the three tests that check them). No behaviour changes.
…iter A unit of one kind and depth that fits within its bound beside what the kind and depth holds is exempt at step 1 of the admission rule, so a reconsideration grants it past an earlier waiter of the same kind and depth that does not fit. The liveness argument's step 2 needs the opposite: no unit of a kind and depth is granted while an earlier one waits. Today that holds only because the walk has one producer per kind and depth, which awaits each admission before asking for the next. This adds a ledger-level point test of the first-come, first-served order: a 40 B unit asked behind a waiting 70 B unit of the same kind and depth must wait, though it fits the bound beside what is held, and must be admitted only after the 70 B unit. It fails on the current rule, which grants the 40 B unit at once; the next commit makes admission first-come, first-served within each kind and depth.
Previously, a reconsideration decided every waiter in turn, so a later unit that fit beside what its kind and depth held was granted at step 1 past an earlier waiter of the same kind and depth that did not fit. The liveness argument's step 2, that no unit of a kind and depth is granted while an earlier one waits, held only by a premise on every caller: one producer per kind and depth, awaiting each admission before asking for the next. A second producer, or code that called Ledger::admit for several units before awaiting the first, could keep the kind and depth filled with small units and starve the earlier waiter. Now admit_waiting keeps, per critical section, the kinds and depths at which it has refused a unit (for room or for its depth's replies) and leaves every later waiter of those kinds and depths queued, undecided. The fast path in Admit::queue already decides only when nothing waits. So admission is first come, first served within each kind and depth, and step 2 holds by the rule. Across kinds and depths the order is unchanged: deeper first, with step 1 bypassing the closed budget and the share, which is what resolves the deeper-first closure and the hold-and-wait schedules. A skipped waiter leaves the budget open: it waits for the earlier unit, which closed the budget already if it waits for room. decide is unchanged. In the walk, at most one unit of a kind and depth waits at a time, so the skip never fires and every session behaves as before. The admission module's docs state the clause in the rule section and place the decision order in admit_waiting and Admit::new beside the steps in decide. Premise (P6) shrinks to what the argument still uses, an answer's questions before its next answer, and says a second producer costs parallelism, not liveness. The liveness argument restates step 2 as a property of the rule, steps 3 and 4 in terms of the unit first in line at its kind and depth, and step 5 as an induction along each kind and depth's line. Claim (M) gains a step: whether a waiter is left undecided depends only on earlier outcomes in the critical section, so it agrees at both budgets up to the first difference, and (M) holds as stated. Ledger::admit's doc says a request is live from the call and is admitted in order within its kind and depth; Work::answering_walk's doc says the walk is its level's one producer, and that concurrent reply processing would add waiters, not room. The admission proptest checks the clause after every operation: no unit is granted while an earlier unit of its kind and depth waits. Its checks that a waiter is not within its bound and waits only behind a closed budget now apply to the unit first in line at its kind and depth. The test kit's check that no two units of a kind and depth wait at once is kept; its doc now says it checks the walk's structure, since a second waiter no longer threatens liveness.
The module opens with a plain account of admission built on concrete actors: a stage asks its side's ledger for a grant, the ledger lets each kind and depth hold one bound's worth exempt, charges the budget for the rest, grants first come, first served within a kind and depth, and so keeps memory bounded while every unit is eventually admitted. The rule's four steps, the premises, and the memory, liveness and monotonicity arguments follow, each stated once with every condition it rests on. - Every action has a real actor (a stage, the ledger, `decide`, `admit_waiting`, a charge), never "the walk", "admission", "the rule" or a step. - Restatement, history, measured magnitudes and derivations that change nothing a maintainer does are gone: one "what must stay" list, one boundary-case list, the liveness argument as one induction, claim (M) without numeric counterexamples, and the invitation share reduced to what it limits, its check, the need, the pace and why claims nest. - (P7)'s list of when each part of a grant returns goes; the ledger's module docs, `materialized::work::charges`, the pipe module and the proxy's `Outstanding` say so at the code that returns each part. - Step 2's refusal reads as closing the budget to every later unit of the critical section, as `admit_waiting` does. - Item docs and line comments say one thing each; `Admit::new` gains a `# Panics` section for its depth assertion. The admission proptest's doc states each reason a unit may wait as the test checks it. Premise labels (P1) to (P8), the rule's step numbers, claim (M), and the headings cited by anchor keep their names, so no citing site elsewhere changes. No code changes.
…act on one platform Previously, the justfile, the budget-curves tool and three throughput test docs said each session is an exact function of its pair, link and budget, reproducing exactly on any machine. Rerunning the recorded budget-curve sessions on illumos, from the very tree that recorded them on Linux, gives slightly different heap figures and admission counts, and in one session a duration 269 byte-times longer: the sessions are deterministic per platform, not across platforms. Now each of those docs says the sessions are exact on one platform and can differ slightly across platforms. Only comments change.
The guide now answers, in order, what the budget does, what the default and minimum are, how to choose a budget from the link's bandwidth-delay product and the replicas' size and overlap, what a low or high budget costs, and how far to trust the figures. It leaves the exact contract to Peer::sync_memory_budget and states no internal quantity. Its figures come from the budget curves rendered at fa32b49fe2ef: 601 simulated sessions between replicas of 100,000 and 1,000,000 messages of 100 bytes, sharing none, 90% or 99%, over links of 1 Gb/s with 1, 10 and 100 ms round trips. Each figure for a budget enough to match an unlimited session is either the smallest measured budget from which every larger measured budget ran exactly as long as the unlimited session, or, where no measured budget did, the smallest budget at which the unlimited session's admissions all fit.
plaidfinch
changed the title
plaidfinch
marked this pull request as ready for review
November 8, 2026 05:45
This was referenced Nov 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a huge refactor, and I consider it a "rough cut", in that I want to clean up internal structure, naming, and documentation. However, I am confident enough that It Works that I'm putting it in now, with follow-ups to come. Claude has diligently re-arranged the commits in this PR to maximize reviewability. Each is atomic and reasonably small. I have manually self-reviewed all 269 of them, focusing especially on production code changes.
Without further ado... here's what's new with Rumors:
We replace the multi-stream
Linkabstraction, previously a required piece of boilerplate for all Rumors users, with a single ordered duplex byte stream, a.k.a.AsyncBufRead/AsyncWrite, of the caller's choice. Most of the rest of this is in support of, or enabled by, this. A whole bunch of parts of the protocol get optimized along the way, such as more granular decoding of replies, dramatic reduction in syscalls by using buffering on read and write sides, and tuning of futures-related costs. The public API also sees some minor improvements, thanks to things discovered en route to the finish line.A central premise of Rumors' design is that we must ensure bounded memory use, tunable by the user. However, combined with this, a naive mux/demux applied to the pre-existing protocol would allow the previously isolated per-tree-level streams to interfere with one another, which would result in deadlocks. This is because the prior protocol ensured bounded memory use by exerting back-pressure on the receiving side. Interleaving the streams into one would therefore have resulted in a situation where the receiver refused to process more input (because it was trying to apply backpressure to ensure bounded memory usage), but was also unable to make any more progress, because the very information it needed to do so was buried in the stream it refused to read.
In this PR, we invert this: instead, the sender exerts backpressure on itself by dynamically calculating the maximum possible size of the response which could possibly be induced by its own message, only dispatching that message once it has room in its memory budget for the induced response. Receivers therefore never block on accepting messages from the wire, which (along with a few other conditions) ensures deadlock freedom.
To support this mechanism effectively, we change the accounting mechanism used to enforce the memory cap on synchronization sessions. In the previous protocol, we used a statistical model based on the distribution of uniform hashes throughout the tree to give a bound on the possible response sizes we could expect at any given stage of the protocol. This was computed once per session and used to size bounded channels, which exerted back-pressure on their individual corresponding wire channels when they were too full. This model was extremely complex, and vastly over-estimated the actual memory that a session would usually use — and also is useless for the sender-enforced back-pressure design we were moving to anyhow.
In place of this statistical channel sizing, the new protocol dynamically instruments almost every item allocated within it to track the number of bytes currently in play against its maximum permissible memory budget. This is pretty efficient, because such tracking requires only some atomic integers and a comparatively infrequently-used uncontended mutex. Using this, each side of the protocol can precisely determine when to slow itself down to avoid over-allocating beyond its budget. The only remaining statistical estimation is a running average of each node's child occupancy computed during each stage of the traversal, which is used to more optimistically allow requests than otherwise would be possible.
A significant number of tests are built out in this PR to check that our internal accounting accords with what actually happens in a real allocator, by using
#[global_allocator]to set a custom allocator which tracks tagged regions in our code and attributes allocations to check whether the protocol's own accounting lines up with reality. This mechanism was used to iteratively refine that accounting until it did match reality. Additional tooling that comes along in this PR includes a binary which measures a real connection (or a simulated one, via Linux'stc/netem) to see what kind of slowdown different memory budgets induce in a realistic network.Memory budgeting continues to exist as a kind of "rev limiter" to ensure that Rumors doesn't chew up unbounded memory. It still doesn't account for things like allocator rounding, serialization, or a few internal places where buffers can double — and it doesn't need to, to be an effective approximate bound. Due to the minimal memory required to keep processing moving, the minimum allowed memory budget is about 37 MiB, which, compared to "unlimited", costs about 5% or less slowdown on very fast links (< 1 ms), but 800% or more on higher-latency ones. The default memory budget is 512 MiB, which costs essentially no slowdown on any realistic link for set sizes up into the tens of millions, and theoretically beyond. Most gossip rounds will see nothing close to their budget. Surprisingly, pessimal cases are not when a node first bootstraps into the network and catches up on everything — they are when there is large mutual divergence between two already established nodes, for example if a node goes offline for a prolonged period of time, or a network partition suddenly resolves.
Separately to all the above, this protocol change also decreases the Merkle hash width to 16 bytes (128 bits), since this is all we need to prevent accidental collisions (128 bits is good enough for a UUID, after all). Naively, this is still perfectly safe because hostile peers are out of the security model for the crate; we assume that all peers are authorized, and that the channels between them are authenticated. However, an adversary outside the gossip network could still "steer" its contents indirectly by either controlling its topology (severing links, etc.), or injecting messages into the network at chosen nodes by, e.g. interacting with an application which generates events in the gossip network. An attacker with the capability to grind 16-byte hashes offline could, in principle, use a network position to engineer a hash collision, and thus corrupt the integrity of the gossip network. I should be clear that the offline attack is still hard, and what's more, the gossip network needs to stay either perfectly predictable or static during the time of the offline precomputation — one message the attacker doesn't anticipate invalidates their entire hash-grind. But it is still possible in the old model, and so we fix it here.
The way we fix this: each network's public ID is now derived from a "network secret", the root of a (very small) key hierarchy. This network secret is never revealed in cleartext via the Rust API or over the wire. From the library user's perspective, nothing changes: the network ID is still a random 128 bits. But under the hood, the secret network key from which the public ID is derived is also used to derive two subkeys, which themselves are used as inputs to keyed hashes. This means that instead of the path to a leaf being a pure function of its
Version, it's now a (non-invertible) function of the (public)Versionand the network secret; similarly for the Merkle hash at every internal node. This makes it impossible for an attacker who has never been part of the gossip network to perform an offline precomputation attack targeting hash collisions, because they cannot predict any hash without the network secret. The network secret cannot be rotated for the lifetime of the gossip network; we accept this risk, because the risk it mitigates is esoteric at best. If the attacker is in a position to steal a network secret from a peer, they must either have compromised a peer or once been a peer — in which case, they could have directly corrupted the gossip set via the protocol instead of through this collision attack.Newly bootstrapping nodes are provided the network secret via bog-standard one-shot HPKE, relying on the authentication of the connection between peers to prevent man-in-the-middle attacks on this exchange. This design was chosen specifically so that integrity of the gossip set rests solely on integrity of the transport. While standard deployments will likely use an authenticated and confidential transport (i.e. (m)TLS), exotic authenticated-but-non-confidential transports give up only confidentiality of the gossip set, and do not invite any additional attack on its integrity.