Skip to content

Use u64 trie iteration tokens#52

Open
MesTTo wants to merge 1 commit into
Adam-Vandervorst:masterfrom
MesTTo:pr/pathmap-u64-iter-tokens
Open

Use u64 trie iteration tokens#52
MesTTo wants to merge 1 commit into
Adam-Vandervorst:masterfrom
MesTTo:pr/pathmap-u64-iter-tokens

Conversation

@MesTTo

@MesTTo MesTTo commented Jul 4, 2026

Copy link
Copy Markdown

Summary

Answers the question the new_iter_token doc comment already asks: the
token is a node-local cursor, so it never needs to encode a whole path, and
64 bits are enough. The ByteNode stops caching the remaining mask word
inside the token — the source of the 128-bit requirement — and keeps only
one more than the last byte returned, recomputing the next set bit from its
own mask (next_iter_byte_from: a masked trailing_zeros probe across at
most four words, still O(1) per step). Every other node type already used
small counters, so this is purely a type change for them.

iter_token_for_path semantics are unchanged (resume strictly after the
given byte); the word-edge cases (63/64, 127/128, 191/192, 255) are pinned by
a new test. Consumers (the zippers, cursors, merkleization, viz) treat
tokens as opaque, so the swap is type-level there.

Result

ReadZipperCore shrinks from 144 to 128 bytes (measured on both this branch
and master), and each ancestor-stack entry loses 8 bytes of padding.
Interleaved A/B against master (two reps each side): superdense_k_path
-7..-8%, sparse_iter -23..-47%, binary_iter up to -53%, no regressed
cell in either bench. The resume-strictly-after contract is also modeled in
Alloy (fac27_iter_token in a separate verification repo), UNSAT in scope.

Test plan

  • cargo test --release --lib (661/0, matches the master baseline)
  • cargo miri test on the new boundary test and the touched dense-node tests
  • cargo test --release --doc (7/0)
  • cargo bench --bench superdense_keys / sparse_keys / binary_keys -- iter

Implements the answer the iter-token note in the TrieNode trait already
sketched: the token is a node-local cursor, so it never needs to encode a
whole path, and 64 bits are enough. The ByteNode stops caching the remaining
mask word inside the token (the source of the 128-bit requirement); it keeps
only one more than the last byte returned and recomputes the next set bit
from its own mask (next_iter_byte_from, four masked trailing_zeros probes,
still O(1) per step). All other node types already used small counters.

iter_token_for_path semantics are unchanged (resume strictly after the given
byte; the 63/127/191/255 word-edge cases land identically through the word
advance), pinned by a new mask-word-boundary test. Consumers treat tokens as
opaque, so the swap is type-level for them; ReadZipperCore shrinks from 144
to 128 bytes (measured on both trees), and each ancestor stack entry loses 8
bytes of padding.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant