fix(recall): bound the embedding call so one slow upstream cannot hold recall open - #38
Merged
Merged
Conversation
…d recall open generateEmbedding had no timeout. Hybrid recall awaits it, so a single slow upstream call held the whole request open for as long as the network took. Measured over a 14-day recall log (2404 calls): query_path=sync n=844 slow(>=3s)=0 avg 9ms query_path=strict n=144 slow(>=3s)=0 avg 60ms query_path=hybrid n=1416 slow(>=3s)=282 avg 2527ms Every slow call was on hybrid — the only path that embeds. Overall p50 is 327ms and p90 is 10635ms, which is not a tail, it is a second mode. The hybrid distribution has a cliff with an empty middle: p50 501ms · p70 919ms · p80 2290ms | p85 10695ms · p90 10734ms · p95 10880ms Nothing lives between ~3s and ~10s. That is the shape of an upstream stall, not of our work getting slower under load. ## Not the diagnosis we had This was on the books as "the resident server has a ~10s cold start, go find out where the server spends it". Two things kill that reading: - idle does not predict it. Median gap before a slow call is 3 minutes, before a fast one 1 minute; 16% of slow calls follow a >=10min gap vs 11% of fast ones. - the resident server is not where the time goes. It is an outbound HTTP call with no deadline, on the hot path, inside a Promise.all. The waiting also bought nothing. FTS runs in that same Promise.all and is synchronous, so its rows were already in hand — the request just sat on the socket holding them. ## Fix AbortSignal.timeout, default 2500ms, EMBEDDING_TIMEOUT_MS to override. 2.5s cuts the entire stall cluster (282 of the 286 calls it touches) at the cost of one legitimately slow embed; tightening to 1s would eat 84 more real ones and gain nothing, since the cluster is already gone by then. Resolved per call rather than at module load. Capturing env at import time makes the knob untunable by anything that configures itself after the import — which is every embedder of this library, and every test. Same trap as the DB path. ## Degradation stays visible A timeout returns null, the vector leg is skipped, and RRF proceeds on FTS rows. That is a quality degradation, so it is labelled rather than swallowed: the result carries _degradedTo / _degradeReason = 'embedding-timeout', matching what the early-bail branch already sets, and the timeout logs as its own line instead of reading like a broken upstream. Fast-and-quietly-worse is the failure mode this is guarding against. generateEmbedding is now exported. The timeout is only testable through it — going via recallMemoriesHybrid cannot discriminate on a machine without the sqlite-vec extension (CI), because that path early-bails to FTS before it ever embeds and would pass either way. ## Verification Red-then-green with the timeout removed: the stalled call runs 30024ms against a 30s stall — it waits the whole thing out — and returns an object instead of null with no degradation reason. Exactly those three assertions fail; both healthy-path assertions stay green, so they are not coupled to the fix. Full suite green: embedding-timeout 7 · cold-pool 6 · recall-contract 14 · ranking-importance 5 · recall-endpoint 17 · query-rewrite 6 · memory-health 68 · locations 51 · provenance 25 · encoding-damage 18 · injection-hygiene 14 · level-migration 10 · supersede-shrink 24 · anchor-pinned 4. [prediction] 修复: hybrid 的第二个模态(~10.6s 那簇, 占 20%)消失, p90 从 10635ms 掉到 2500ms 量级; 发起方多数在 1500ms 就放弃了, 那部分白烧的服务端时间一起省掉 [prediction] 风险: 那 20% 的调用改为只有 FTS 结果, 排序质量下降。如果开始出现 "召回变笨了", 先看 _degradeReason='embedding-timeout' 的比例——比例高说明该治 上游连通性(代理/DNS), 不是把超时调回去 [prediction] 验证: 观察期后 hybrid 的 p85/p90 应落在 2500ms 附近而不是 10.6s; 且带 embedding-timeout 标记的调用数应约等于原先 >=3s 的调用数 Co-Authored-By: 千夏 <qianxia@clawgamers.com>
MXAntian
force-pushed
the
fix/bound-the-embedding-call
branch
from
September 1, 2026 14:54
ce2275b to
ed04c25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
generateEmbeddinghad no timeout. Hybrid recall awaits it, so a single slow upstream call held the whole request open for as long as the network took.Measurement
14-day recall log, 2404 calls:
syncstricthybridEvery slow call is on
hybrid— the only path that embeds. Overall p50 = 327ms, p90 = 10635ms. That's not a tail, it's a second mode.The hybrid distribution has an empty middle:
Nothing lives between ~3s and ~10s. That's the shape of an upstream stall, not of our work getting slower under load.
This wasn't the diagnosis we had
It was on the books as "the resident server has a ~10s cold start, go find out where the server spends it." Two things kill that reading:
Promise.all.The waiting also bought nothing: FTS runs in that same
Promise.alland is synchronous, so its rows were already in hand. The request just sat on the socket holding them.Fix
AbortSignal.timeout, default 2500ms,EMBEDDING_TIMEOUT_MSto override.2.5s cuts the entire stall cluster — 282 of the 286 calls it touches — at the cost of one legitimately slow embed. Tightening to 1s would eat 84 more real ones and gain nothing, since the cluster is already gone by then.
Resolved per call, not at module load: capturing env at import time makes the knob untunable by anything that configures itself after the import — which is every embedder of this library, and every test. Same trap as the DB path.
Degradation stays visible
A timeout returns null, the vector leg is skipped, RRF proceeds on FTS rows. That's a quality degradation, so it's labelled rather than swallowed:
_degradedTo/_degradeReason = 'embedding-timeout', matching what the early-bail branch already setsFast-and-quietly-worse is the failure mode this guards against.
generateEmbeddingis now exported. The timeout is only testable through it — going viarecallMemoriesHybridcan't discriminate on a machine without the sqlite-vec extension (CI), because that path early-bails to FTS before it ever embeds and would pass either way.Verification
Red-then-green with the timeout removed:
It waits out the entire 30s stall. Exactly the three assertions that matter fail; both healthy-path assertions stay green, so they aren't coupled to the fix.
Full suite green: embedding-timeout 7 · cold-pool 6 · recall-contract 14 · ranking-importance 5 · recall-endpoint 17 · query-rewrite 6 · memory-health 68 · locations 51 · provenance 25 · encoding-damage 18 · injection-hygiene 14 · level-migration 10 · supersede-shrink 24 · anchor-pinned 4.
Risk
That 20% of calls now returns FTS-only results — worse ranking, same answer. If recall starts feeling dumber, check the share of calls carrying
_degradeReason='embedding-timeout'first: a high share means fix upstream connectivity, not raise the timeout back.