Summary
Retention no longer does store-sized work inside its write transactions (#2646). Every maintenance tick (about every 5 minutes) still reads store-sized data on the reader, though:
- Payload GC reference closure (
tracedecay_lcm::gc::read_payload_gc_snapshot). A payload reference can sit in the free text of any raw row (content, snippet_text, index_text, metadata_json), so the closure reads every raw message's text on each tick.
- Observation-retention candidate selection (
crates/tracedecay-global-db/src/observation/retention.rs, anchor/observation/provenance passes). Each walks retrieval_anchors, the binding and disposition joins, and observation_repository_provenance, then filters by the released-disposition predicate. It reads the fat rows before it knows any is eligible.
Evidence (isolated profile, 1,200-transcript synthetic corpus, project sessions.db 1.65 GB, about 19k project messages)
Hotpath over two ticks:
| span |
avg |
sessions.lcm.gc.snapshot |
6.69 s (p95 8.30 s) |
global_db.observation.retention |
4.63 s |
Before #2646, one tick's attributed reads were:
On the operator's 16.8 GB sessions.db the closure alone is several GB per tick.
Expected
The per-tick read follows what changed since the last tick:
- GC: a durable reference index maintained where references are written, or candidates recorded where references are removed.
- Observation retention: select from the disposition ledger (released states past the window) instead of the evidence tables.
The typed GC mark handshake from #2646 (upsert_payload_metadata clears the mark) already makes an incremental closure safe to act on. Refs #2521.
Summary
Retention no longer does store-sized work inside its write transactions (#2646). Every maintenance tick (about every 5 minutes) still reads store-sized data on the reader, though:
tracedecay_lcm::gc::read_payload_gc_snapshot). A payload reference can sit in the free text of any raw row (content,snippet_text,index_text,metadata_json), so the closure reads every raw message's text on each tick.crates/tracedecay-global-db/src/observation/retention.rs, anchor/observation/provenance passes). Each walksretrieval_anchors, the binding and disposition joins, andobservation_repository_provenance, then filters by the released-disposition predicate. It reads the fat rows before it knows any is eligible.Evidence (isolated profile, 1,200-transcript synthetic corpus, project
sessions.db1.65 GB, about 19k project messages)Hotpath over two ticks:
sessions.lcm.gc.snapshotglobal_db.observation.retentionBefore #2646, one tick's attributed reads were:
session_summary_sources)On the operator's 16.8 GB
sessions.dbthe closure alone is several GB per tick.Expected
The per-tick read follows what changed since the last tick:
The typed GC mark handshake from #2646 (
upsert_payload_metadataclears the mark) already makes an incremental closure safe to act on. Refs #2521.