memory: charge or release retained parse trees, regex caches, and fixed runtime heap

[ScriptedAlchemy]

Summary

Follow-up to #2490. Retained-heap attribution of an isolated daemon shows which live Rust/C heap no resident owner charges. #2490's refresh starvation and the missing swap term are fixed separately; the components below are still uncharged and not released by any owner.

Evidence

Isolated daemon (debug tracedecay/production build with the system allocator so heaptrack sees every allocation), MemoryMax=6G, MemorySwapMax=1G, isolated HOME/TRACEDECAY_DATA_DIR. Corpus: depth-1 clone of master at cc5a895 (6,684 files). The snapshot was taken after the cold index sealed generation 1 and one refresh commit had been attempted (and refused on master). heaptrack 1.5 --raw; live bytes are counted at the kill point, grouped by call site.

Component (live at snapshot)	Bytes	Allocations	Owner today
Privacy credential regex lazy-DFA caches (one per regex per scanning thread)	201,597,261	201,330	none
Tree-sitter trees in the per-worktree SharedRetainedParsePool (256 docs, bound is on source bytes only)	181,191,207	1,868,404	none
Tool / MCP / SDK catalog and schema caches (process-lifetime)	130,039,371	996,287	none (fixed)
Captured source byte pool (SharedCodeIndexBytePoolV1)	101,735,806	51,887	ledger (snapshot charge)
Canonical serializer / extraction rows	29,945,854	4,305	none
SQLite page cache, schema, statements	21,142,019	126,815	none
Sealed manifest / retained text generation	12,074,505	119,590	none
Other	2,584,369	16,829
Total live heap	680,384,120	3,385,448

At that point the same process had RssAnon 2,423,068 kB plus VmSwap 1,037,368 kB, and no resident owner was charged. heaptrack's own trace tree (27.1M nodes) inflates the RSS, so the gap between RSS and live heap there is mostly heaptrack plus allocator retention. It is not live Rust state.

Production (mimalloc) debug build from the fix branch, same corpus and cap, cold index plus one admitted refresh (generation 2 sealed at the refresh commit): RssAnon 3,306,024 kB, VmSwap 1,048,448 kB, admission sample 4,483,579,904 B. Charged owners were decoded_generation 2,760,435,008 B, which leaves ≈1.6 GB above the charged owners.

What remains

1. Retained parse trees: per worktree, bounded by document count, not charged. There is no truthful byte measure for tree-sitter trees today. Either (a) register the pool as a resident owner with a structural tree-bytes bound (node count × subtree layout) and release it idle or under pressure, or (b) stop retaining trees parsed by a cold full build. Only changed-file rebuilds benefit from retention.
2. Regex caches: bounded by rules × concurrent scanners × the lazy-DFA cache capacity (2 MiB default), never released. Needs a measured decision: a bounded cache capacity or a shared scanner width. Hotpath before/after on the privacy scan is required before changing it.
3. Fixed runtime allowance: the catalog caches (130 MB), SQLite, and allocator overhead need an allowance defined by the resident-memory authority. Then the charged-owners-plus-allowance invariant can be asserted against measured anon RSS.
4. Graph activation under load: in the debug build on a loaded host, graph publication probe deadline exceeded retries forever while the 2.76 GB decoded generation stays held for the retry. code_graph_serving stays pending.

[ScriptedAlchemy]

#2592 (squash 0e4a5ab) removes the three largest uncharged retentions instead of charging them. Heaptrack on the same journey as this issue (isolated daemon, 6,697-file clone at 161eccf, cold index plus one refresh commit, system allocator, 6G cap):

Component	master	after #2592
Retained parse trees	235,150,946	673,424
Credential regex caches and compiled rules	201,401,355	6,964,649
Catalog and schema caches	130,021,379	67,756,892
Captured source byte pool (ledger-charged)	101,989,982	101,977,390
Serializer / extraction rows	27,932,449	33,068,150
SQLite	22,635,374	22,567,483
Sealed manifest / retained text generation	12,100,742	12,792,611
Total live	734,102,454	248,467,929

• Parse trees (item 1): full builds no longer retain. On master, an increment re-extracting 367 files after a cold build of 6,555 files reparsed 2 from retained trees (0.54% hit rate). Increments still retain what they re-extract.
• Regex caches (item 2): per-ruleset explicit regex_automata caches in a thread-affine pool; bulk capture holds a batch whose caches are freed when it ends. Throughput is equal to master within noise at 1, 8, and 32 threads (numbers in perf(memory): stop retaining scan caches, parse trees, catalogs #2592). Retained after a 32-thread capture: 15.9 MB vs 599 MB.
• Catalogs (part of item 3): one process-wide schema-bearing snapshot instead of four copies; the discovery cache stores only the payload.

Production (mimalloc) journey, generation 2 sealed plus about 2 minutes: master was 1,601,212,999 B above charged owners, #2592 is 1,076,560,455 B above.

Still open: the fixed runtime allowance and the #2490 invariant. Live uncharged heap is now about 146 MB: catalogs 67.8 MB, serializer scratch 33.1 MB, SQLite 22.6 MB, retained text generation 12.8 MB, compiled rules 7.0 MB, other 2.6 MB. The remaining ~0.93 GB of the RSS gap is not live Rust heap. It is allocator retention plus graph activation work that never finishes on the loaded host (#2505). An allowance fitted to that gap would not be a fixed allowance. Deriving one needs a daemon that settles after activation, with allocator retention split from live state. Serializer scratch and the retained text generation are per-worktree and still uncharged.

[ScriptedAlchemy]

The graph owners' charges are now truthful (#2643). The remaining uncharged Rust heap and the allowance question are measured in #2490 (latest comment): the gap is about 630 MB fixed plus about 75 KB per indexed file, and almost all of it sits in mimalloc arenas. So this stays open until the proportional owners are charged.

[ScriptedAlchemy]

#2661 (154a0eb) returns idle index workers' heaps (233 MB at 296 files on 36 workers) and deletes the retained captured source (16 KB per file). The full attribution and before/after gaps are in #2490 (latest comment). This stays open: the remaining uncharged heap is still proportional. It is about 43 KB per file of mimalloc pages with no live data under the owners' data and 16.5 KB per file of glibc heap. The increment retained parse pool is also uncharged: 12.2 MB of tree-sitter parser stacks plus about 6 MB of artifacts for 38 files. The fixed parts are catalogs at 67 MB and SQLite at about 21 MB.

[ScriptedAlchemy]

#2727 (f5b938b): full numbers in #2490 (latest comment).

• Tokio and SQLite threads: they now return freed heap when they idle. Per-thread collect at 293 files went from 11.1 to 2.1 MB (tokio) and 5.2 to 0.1 MB (SQLite).
• Serializer pools: released when their thread idles (8 × 1 MiB object buffers + 64 KiB digest buffer per thread).
• Parse-tree cache: retained documents keep only their tree. Parser stacks are gone: 78,976 B across 8 test documents, 12.2 MB across 38 refreshed documents in fix(memory): free idle worker heaps and built source bytes #2661's run.
• Graph owners: built in their own heaps and charged their pages. The catalog's pages are 1.6× its structural bytes. Settled full-clone gap: 716.8 → 298.6 MB.

Still open, tracked in #2706: the retained parse pool's trees and artifacts are uncharged, and so is the build residue left when the owners are released. The catalogs (67 MB) and SQLite's heap (21 MB) are fixed, but with SQLite memory statistics off (#2136) they have no live measure.

[ScriptedAlchemy]

Update in #2490 (latest comment). #2773, #2775 and #2778 removed the glibc arenas, charge the retained parse pool, and stop dead weak cache entries pinning captured sources. The gap is now flat across 336 to 1,345 files, at 154–178 MB. A heavy-file refresh still adds about 210 MB of allocator retention (#2706), so the fixed allowance is not added yet.