fix(lancedb): 隔离重量级维护并让全新 upsert 批次走 add 快路径

[flyto168]

问题

在线服务进程与重量级 LanceDB 维护（compact/rebuild 等）运行在同一常驻进程内。每日日志的 upsert 走 merge_insert，即使批次里全部是新键，也会对大 fragment 重写数百 MiB 并驻留 GiB 级 Arrow/Lance 原生内存，多次把常驻服务推过内存水位线。

改法

• upsert 先做轻量主键存在性探测：单列字符串主键且批次内全部为新键时，走 LanceDB 仅追加的 add 路径；只有确实包含已存在键的批次才支付 merge_insert 代价（保留上游 PRIMARY_KEY 声明式键不变式，无 by 参数）。
• 维护任务（心跳/重建）可经配置门控，与在线写入路径隔离。

验证

• 定向回归：cascade 套件 259 passed；Ruff check/format 通过。
• 基于上游 main @ a345d08，仅含本主题文件（6 文件）。

范围

仅含通用修复与测试，不含本地部署配置与环境文件。

[gloryfromca]

Thanks for this one, and for #490 and #492 — those two are merged, and the measurements in #492 (the 1/2/4/8 concurrency curve) were good enough to review against directly. This one I'm closing, because the mechanism it describes does not hold on the LanceDB we ship. Details below so the work isn't lost, and so you can reopen on firmer ground if the symptom is still there.

The upsert fast path cannot execute

repository.py gates it on len(key) == 1. Every business table declares a three-column key:

Episode / AtomicFact / AgentCase / Foresight / AgentSkill / UserProfile / KnowledgeTopic
  PRIMARY_KEY = ('app_id', 'project_id', 'id')   len=3

7/7 ineligible. #481 made the key composite on 2026-10-12, a day before this PR, so the branch has never been reachable — the only code that enters it is a test model with PRIMARY_KEY = ("id",).

merge_insert does not rewrite existing fragments

This is the part worth having, because it's the premise.

We ship lancedb 0.34.0, which embeds lance-core 8.0.0 (read from the cargo paths in the compiled _lancedb.abi3.so). In that version's merge_insert.rs, unmatched source rows are routed to new fragments — handle_new_fragments / write_fragments_internal — and existing fragments are touched only through apply_deletions, i.e. only where rows were actually removed.

Observed directly rather than inferred. A 200k-row table whose data lives in one 824 MB file, then one 500-row write:

write	pre-existing files removed	modified	new
add()	0	0	1 file, 2.1 MB
merge_insert, all-new keys	0	0	1 file, 2.1 MB
merge_insert, all-existing keys	0	0	1 file, 2.1 MB

The 824 MB is untouched in every case — including the all-existing one, which I expected to rewrite and which does not. There is no "rewrite hundreds of MiB" path here to avoid.

What does cost more than add is finding the matches: 10.1 ms at 100k rows and 25.0 ms at 400k, against a flat 2.6–3.0 ms for add. That is the full outer join the source describes when no usable index exists — a scan of the three key columns, which is why it scales with table size and why it does not produce GiB of resident memory.

(Related, for whoever picks up #477: an indexed join does exist, but only helps a single-column key — 16.9 → 12.8 ms. Three single-column BTrees on our composite key gave nothing: 24.4 → 25.3 ms.)

The bundled default change costs more than the PR saves

Separately from the fast path, this PR sets optimize_rebuild_interval_seconds to 0.0 in both settings.py and default.toml, disabling the periodic rebuild. That sweep is the only application-level bound on active index UUID growth — rebuild_indexes says so in its own docstring, and so does the constant the PR leaves at 12h, which now disagrees with the config in every production path.

Measured with prune running every fifth beat, 2500-row table with FTS + IVF indexes, counting _indices/:

optimize + prune only (this PR)     beat 25: 59 files → 50: 109 → 75: 159 → 100: 209   (+2/beat, linear)
optimize + prune + rebuild (main)   beat 25:  6 files → 50:   6 → 75:   6 → 100:   6   (flat)

Prune cannot reclaim them; the manifest still references them. At the 60s heartbeat that is roughly 2880 new index files per day per table with nothing collapsing them.

On the symptom

I don't think you imagined it. But the observation that exists is #492's — 8.29 GiB on production replay, with a concurrency curve behind it — and both #490 and #492 are now in main.

A/B on the same 300-file / 3000-entry corpus, real server, RSS sampled once a second:

	drain	peak RSS
cc86c5b (before #490 and #492)	244 s	533 MB
ec32f4b (after both)	114 s	439 MB

2.1× faster, ~18% lower peak, and neither version comes near GiB. That scale is modest — one table at 3000 rows — so it does not prove the symptom can never appear. It does mean I can't see it, on either side of your two merged fixes.

If you can still reproduce it on current main, please reopen with the numbers — the shape of #492's report (peak against a varied parameter) is exactly what makes a memory claim reviewable, and it is what this PR is missing.

Two smaller things, so they aren't lost: ruff format --check fails on orchestrator.py here (CI would be red), and EVEROS_DISABLE_CASCADE_MAINTENANCE also disables prune, which its name does not suggest — if you want that switch, it would be worth its own PR with a runbook entry and a test.

[gloryfromca]

One follow-up, in fairness to the other half of this PR.

My comment above took apart the merge_insert claim but said nothing with data about the rebuild memory claim — the one behind disabling the startup sweep. I've now measured that one too, on a real index with the FTS and vector indexes rebuild_indexes() touches, building and rebuilding in separate processes so the peak belongs to the rebuild:

rows	rebuild	RSS before	peak	delta
20,000	0.04 s	177 MB	270 MB	+93 MB
100,000	0.14 s	182 MB	389 MB	+207 MB
300,000	0.53 s	185 MB	496 MB	+312 MB

On this one your direction was right and I was wrong to wave it off. The transient does scale with table size — roughly 77 MB + 0.78 MB per 1000 rows — and I had previously pointed at the repo's "3–6 MB resident" figure as if it bounded this. It doesn't; that number is the index cache working set, not the rebuild transient.

Where it still doesn't reach the claim: "several GiB" needs on the order of 4 million rows at that slope. If your production store is at that scale, say so and the picture changes — that is exactly the repro condition the PR was missing.

For the record, I tried the same change myself (#507, now also closed) and it went badly: a corrupt state byte crashlooped the server, and the skip silently stretched the rebuild interval to 1.95x. So the idea is harder to land safely than it looks, independent of whose PR it is.

A separate PR for just the startup-sweep question — persisted clock, failing closed, with the data volume stated — would be welcome.