docs(code-index): rank graph rebuild speed cuts - #1578
ScriptedAlchemy wants to merge 2 commits into
Conversation
A retryable graph activation used to erase the prepared serving candidate, and an unfinished clone-fingerprint successor withheld the same seat after exact and lexical owners were ready. Keep the candidate in both cases so search can move off the predecessor while graph retries. Co-authored-by: Zack Jackson <ScriptedAlchemy@users.noreply.github.com>
|
|
Closing as superseded. A sibling change proving the same tool landed first as #1562. Evidence from the triage: git diff --stat origin/master...origin/cursor/graph-rebuild-speed-plan-d813 = exactly the 3 files of b80dd58 (registry.rs, registry/mount.rs, registry/seat_swap_tests.rs) and NOTHING else; its ranked plan lives only in the PR body, which itself states "Draft planning branch at b80dd58; do not merge" and "Do not merge anything from this consolidator." Reopen if the landed version misses an assertion this branch carries; name it and it can be ported as a small follow-up. |
2 participants
Ranked graph-rebuild speed plan
Draft planning branch at
b80dd58; do not merge. KeepRECEIPT_TIMEOUT = 90s. Estimates are non-additive and distinguish measured wall time, sampled CPU share, and inferred work reduction.Inventory correction: #1570 merged into
masterat1e454fffon 2026-09-18 00:01 UTC, so it is now the baseline rather than a future step. #1564 and #1565 are also merged and do not touch the sampled graph-rebuild path. The remaining sequence is: measure the #1570 baseline, land #1562 alone, implement the status/wait cut from #1567, then execute the larger #1568 work cuts. Do not merge anything from this consolidator.Evidence boundary: pre-#1562 CI attempts were 96.671s and 92.174s, with 27,419/29,551 status calls, 98,305 symbols, 2,305 clone pages, and a real N/N+1 serving mismatch. On
b80dd58, the next 93.41s failure was not a clean speed receipt:limit: 3produced 18,084 characters, crossed the 15,000-character response cap, and hid top-levelresults/code_generationin a truncation envelope. Withlimit: 1, the first refresh/seat waits completed and the journey next exposed a separate 20.001s reopen wait. The macOS pstack sample at about 26s has no percentages, but consistently places workers in clone-token serde,clone_payload_digests,canonical_json::write_value, andadmit_validated_file_parts.edge_count = 0is fixture truth, not graph progress: attempt 2 had a current, ready graph with that census.MERGED BASELINE — measure perf(index): share clone-body tokens; scan secrets with the DFA #1570 before writing another shared-cost patch. The live beta.40 sample attributed 42% of index-worker cycles to sanitizer PikeVM, 20% to spin/futex, 16% to malloc/free, and only 9% to extraction; a separate window put 31–36% in deep clone-token copies. perf(index): share clone-body tokens; scan secrets with the DFA #1570 shares token streams instead of cloning them (
crates/tracedecay-code-extraction/src/clone_body.rs:61-78;crates/tracedecay-code-index/src/clones.rs:412-443) and translates RE2\b/\Bto DFA-eligible ASCII boundaries (crates/tracedecay-privacy/src/rules.rs:630-669). There is no same-daemon A/B yet. A transparent planning estimate is 1.6–2.1× index-worker throughput if these cuts remove one-half to two-thirds of the sampled 78% shared-cost bucket; that is not an end-to-end claim. Rerun the exact graph-rebuild journey on currentmasterplus fix(code-index): seat text through retryable graph activation #1562 and replace this estimate with elapsed/CPU/RSS/I/O evidence.LAND NEXT, IN fix(code-index): seat text through retryable graph activation #1562 ONLY — seat the prepared generation through retryable activation. Keep fix(code-index): seat text through retryable graph activation #1562 limited to candidate preservation and owners-ready seating at
crates/tracedecay-code-index-runtime/src/code_index_scheduler/registry/mount.rs:1641-1681,1711-1722. This changes no throughput (1.00× service-demand speedup) but conditionally removes at least one activation backoff (50ms in this test, 30s in production) and, in the observed N/N+1 race, changes “never seats within 90s” to “seat after the current activation attempt.” Do not fold status, polling, clone codec, fixture, or graph-projector work into this PR.FIRST FOLLOW-UP — implement docs(code-index): serving-current latency after seat keep #1567's status-wake cut and repair the receipt observer.
project_code_index_generation_census_readercurrently reuses graph query admission atcrates/tracedecay-code-index-runtime/src/project_reads.rs:85-139,252-300; one status call can enterready_without_stat, capture Git metadata, and post a worker wake atcrates/tracedecay-code-index-runtime/src/code_index_scheduler/registry/serving_reads.rs:888-930,1240-1310. Add a census-only exact-scope read that copies statistics from the seated graph-ready owner and never probes Git or wakes reconciliation. Replace the receipt'syield_now()loops (crates/tracedecay/tests/transport_acceptance_suite/graph_rebuild_status_test.rs:148-173,188-223) with the existing subscribe-before-probe pattern aroundsubscribe_serving_generation_changes(registry.rs:2599-2613; sends atregistry/mount.rs:1932-1938,2042-2070), ending with one public status/search proof. In that test's search helper (graph_rebuild_status_test.rs:114-128), uselimit: 1and fail immediately on a truncation envelope (crates/tracedecay-mcp/src/tools/render.rs:94-151) rather than spinning for 90s. Expected reduction: tens of thousands of status RPCs to O(state transitions), over 99.9% poll-work reduction; estimated end-to-end gain 1.1–1.3× if polling consumed roughly one of four workers, potentially more only if repeated wakes caused extra passes. Do not multiply loop cadence by call count as saved wall time. The Opus status-poll lane remains unmerged: re-check its final A/B, but reject its temporary 300s diagnostic override.FOLLOW-UP AFTER REBASE — retain the non-overlapping measured cuts from #1577. perf(clones): decode clone tokens in one pass; validate each payload once #1577 was built on pre-perf(index): share clone-body tokens; scan secrets with the DFA #1570 fix(code-index): seat text through retryable graph activation #1562 and overlaps
clone_body.rs, so it must be rebased onto currentmasterand remeasured rather than landed as-is. Its one-pass token decoder reduced 686→556µs/file (19%), while its independent census change reduced 98,304 payload validations to 128 (about 1.2s/artifact) atcrates/tracedecay-query/src/retrieval/lexical/projection/artifact/clone_census.rs:27-80. Pre-perf(index): share clone-body tokens; scan secrets with the DFA #1570 expected journey gain was 2.5–3.0s, about 1.03×; the post-Arc token benefit is unknown, while the unique-payload census benefit should remain. Preserve malformed-wire and orphan/mismatched-payload refusals, and confirm the two reported runtime-test failures reproduce on the new baseline.FOLLOW-UP FROM docs(code-index): plan graph-rebuild receipt speed #1568 — represent below-threshold clone bodies compactly before rename/digest/serde work. perf(index): share clone-body tokens; scan secrets with the DFA #1570 removes deep token copies, but the fixture's one-line bodies are still classified below the 30-token floor only after extraction (
crates/tracedecay-code-extraction/src/clone_body.rs:138-184,226-239), then receive a full payload atcrates/tracedecay-code-index/src/chunks.rs:810-860and are serialized again atcrates/tracedecay-code-index/src/production/lexical_page_source.rs:3321-3437. Add a lightweight excluded-body row retaining occurrence, span, token count, andExcludedTooSmall, sotracedecay_similarstays truthfully typed while rename normalization, retained token vectors, payload digests, and successor payload/fingerprint rows are omitted. First assert all 98,304 generated bodies take that branch. Comparable 88k–101k-symbol runs attributed 29.0–42.7s to clone-enabled artifact ingest (benchmark_data/index-bench/clone-envelope-20260916-operating-envelope.json:7-14,76-84,174-182); before perf(index): share clone-body tokens; scan secrets with the DFA #1570 that implied 1.3–1.8× artifact-ingest potential. Treat it as an upper range now and remeasure because perf(index): share clone-body tokens; scan secrets with the DFA #1570 already removed part of that cost.FOLLOW-UP FROM docs(code-index): plan graph-rebuild receipt speed #1568 — validate and decode clone payloads once, then remove the second sealed-source walk.
admit_validated_file_partscalls fullartifacts.validate()on both walks (lexical_page_source.rs:3321-3338), re-running three or four hashes per body throughcrates/tracedecay-code-index/src/chunks/artifacts.rs:322-337andcrates/tracedecay-code-index/src/clones.rs:292-353,515-536. Mint a private validated-segment proof before using structural validation on later admission; this removes 589,824–786,432 canonical hashes for the fixture. Generalize the existing streaming identity-restoring deserializer (crates/tracedecay-code-index/src/production/partitioned_codec.rs:1489-1627) for V3 file markers instead of parse→canonical rewrite→second parse (partitioned_codec.rs:1310-1389), then hand bounded encoded clone pages to the successor rather than reopening every segment atcrates/tracedecay-code-index-runtime/src/code_index_scheduler/serving.rs:3166-3217. Decode is already parallel atpartitioned_codec.rs:2678-2736; do not add nested parallelism or retain all 98k bodies outside the 128MiB ledger. Current measured components support only 1.04–1.15× expected journey gain; larger claims require a fresh profile after perf(index): share clone-body tokens; scan secrets with the DFA #1570.FOLLOW-UP FROM docs(code-index): plan graph-rebuild receipt speed #1568 — reuse prepared bytes and the V16 batched successor writer. The scheduler serializes payload and occurrence only to measure scratch at
serving.rs:3330-3374; the successor serializes them again, reserializes payload for collision comparison, and issues per-body SQL atcrates/tracedecay-query/src/retrieval/lexical/projection/artifact/clone_successor.rs:433-529. Return encoded sizes from the writer and port the 32-row inserts, 512-digest conflict checks, and staged fingerprint sort fromartifact/builder.rs:3780-3997. That changes roughly 294,912 body statements to about 6,144 inserts plus about 192 conflict reads (about 48× fewer SQL calls). Eligible-corpus fingerprint staging measured 2.1GiB→0.4GiB (5.25× less physical write volume); expected wall gain on this relation-free, below-threshold fixture remains only 1.00–1.05× because SQL was not sampled hot and it emits no fingerprints.FOLLOW-UP FROM docs(code-index): plan graph-rebuild receipt speed #1568 — prioritize a due graph retry over optional clone drain. A retained clone task can start at
registry/mount.rs:581-661and is joined atregistry/mount.rs:2221-2272before the next graph retry. Gate successor-only drain while graph activation is pending; resume after graphReadyor terminal refusal. This removes0..remaining clone-tailfrom retry latency but no service demand, so the speedup is conditional and currently unmeasured.NO SPEED CHANGE — accept test(code-index): classify graph-rebuild fixture as relation-free #1575's fixture verdict and do not implement incremental semantic edges here. #1575 proves the 98,305 independent functions have no calls/imports and therefore seal zero canonical semantic edges. #1572 proves the same at a lower-level source shape; do not treat the two test PRs as two performance fixes.
edge_countcomes fromedges.len()atcrates/tracedecay-code-index/src/production/mod.rs:2164-2176, while graph activation still emits roughly 98k symbol entities and file-symbol relations atcrates/tracedecay-code-index/src/graph_projection/builder.rs:136-399. Estimated incremental semantic-edge speedup for this fixture: 1.00× / N/A. If a later edge-heavy Hotpath run makes graph projection dominant, keepNameResolvededges behind the complete-roster barrier atproduction/helpers.rs:380-438and all draft rows unservable until full digest proof and verified-head CAS.Execution gate: run the exact production journey with unchanged 90s and 20s bounds after each sequence point; keep cold refresh, one-file reopen, and unchanged reopen timings separate. Capture OS elapsed/CPU/RSS/I/O plus narrow Hotpath spans, preserve byte-identical sealed/text/graph digests, and reject gains that add swap/RSS pressure or foreground latency. Completion requires one non-vacuous exact pass with no timeout, budget, worker-cap, allocator, or retry-floor change.