Skip to content

fix(recall): bound the embedding call so one slow upstream cannot hold recall open - #38

Merged
MXAntian merged 1 commit into
mainfrom
fix/bound-the-embedding-call
Sep 1, 2026
Merged

fix(recall): bound the embedding call so one slow upstream cannot hold recall open#38
MXAntian merged 1 commit into
mainfrom
fix/bound-the-embedding-call

Conversation

@MXAntian

@MXAntian MXAntian commented Sep 1, 2026

Copy link
Copy Markdown
Contributor

generateEmbedding had no timeout. Hybrid recall awaits it, so a single slow upstream call held the whole request open for as long as the network took.

Measurement

14-day recall log, 2404 calls:

query_path n slow (≥3s) avg
sync 844 0 9ms
strict 144 0 60ms
hybrid 1416 282 2527ms

Every slow call is on hybrid — the only path that embeds. Overall p50 = 327ms, p90 = 10635ms. That's not a tail, it's a second mode.

The hybrid distribution has an empty middle:

p50  501ms · p70  919ms · p80 2290ms  |  p85 10695ms · p90 10734ms · p95 10880ms

Nothing lives between ~3s and ~10s. That's the shape of an upstream stall, not of our work getting slower under load.

This wasn't the diagnosis we had

It was on the books as "the resident server has a ~10s cold start, go find out where the server spends it." Two things kill that reading:

  • Idle doesn't predict it. Median gap before a slow call is 3 min, before a fast one 1 min. 16% of slow calls follow a ≥10min gap, vs 11% of fast ones. No correlation.
  • The server isn't where the time goes. It's an outbound HTTP call with no deadline, on the hot path, inside a Promise.all.

The waiting also bought nothing: FTS runs in that same Promise.all and is synchronous, so its rows were already in hand. The request just sat on the socket holding them.

Fix

AbortSignal.timeout, default 2500ms, EMBEDDING_TIMEOUT_MS to override.

2.5s cuts the entire stall cluster — 282 of the 286 calls it touches — at the cost of one legitimately slow embed. Tightening to 1s would eat 84 more real ones and gain nothing, since the cluster is already gone by then.

Resolved per call, not at module load: capturing env at import time makes the knob untunable by anything that configures itself after the import — which is every embedder of this library, and every test. Same trap as the DB path.

Degradation stays visible

A timeout returns null, the vector leg is skipped, RRF proceeds on FTS rows. That's a quality degradation, so it's labelled rather than swallowed:

  • result carries _degradedTo / _degradeReason = 'embedding-timeout', matching what the early-bail branch already sets
  • the timeout logs as its own line instead of reading like a broken upstream

Fast-and-quietly-worse is the failure mode this guards against.

generateEmbedding is now exported. The timeout is only testable through it — going via recallMemoriesHybrid can't discriminate on a machine without the sqlite-vec extension (CI), because that path early-bails to FTS before it ever embeds and would pass either way.

Verification

Red-then-green with the timeout removed:

✓ the stalling endpoint was actually reached (fixture is wired up)
✗ a stalled embedding gives up on schedule — took 30024ms, budget 600ms, stall 30000ms
✗ a timed-out embedding returns null rather than throwing — got object
✓ a healthy embedding still comes back
✓ a healthy embedding is not delayed by the timeout machinery
✓ recall still returns rows — it degrades, it does not fail
✗ a timeout is recorded as its own degradation reason — _degradeReason=undefined
FAIL: 4 passed / 3 failed

It waits out the entire 30s stall. Exactly the three assertions that matter fail; both healthy-path assertions stay green, so they aren't coupled to the fix.

Full suite green: embedding-timeout 7 · cold-pool 6 · recall-contract 14 · ranking-importance 5 · recall-endpoint 17 · query-rewrite 6 · memory-health 68 · locations 51 · provenance 25 · encoding-damage 18 · injection-hygiene 14 · level-migration 10 · supersede-shrink 24 · anchor-pinned 4.

Risk

That 20% of calls now returns FTS-only results — worse ranking, same answer. If recall starts feeling dumber, check the share of calls carrying _degradeReason='embedding-timeout' first: a high share means fix upstream connectivity, not raise the timeout back.

@MXAntian
MXAntian marked this pull request as ready for review September 1, 2026 14:41
@MXAntian MXAntian closed this Sep 1, 2026
@MXAntian MXAntian reopened this Sep 1, 2026
…d recall open

generateEmbedding had no timeout. Hybrid recall awaits it, so a single slow
upstream call held the whole request open for as long as the network took.

Measured over a 14-day recall log (2404 calls):

  query_path=sync    n=844   slow(>=3s)=0    avg     9ms
  query_path=strict  n=144   slow(>=3s)=0    avg    60ms
  query_path=hybrid  n=1416  slow(>=3s)=282  avg  2527ms

Every slow call was on hybrid — the only path that embeds. Overall p50 is 327ms
and p90 is 10635ms, which is not a tail, it is a second mode.

The hybrid distribution has a cliff with an empty middle:

  p50 501ms · p70 919ms · p80 2290ms  |  p85 10695ms · p90 10734ms · p95 10880ms

Nothing lives between ~3s and ~10s. That is the shape of an upstream stall, not
of our work getting slower under load.

## Not the diagnosis we had

This was on the books as "the resident server has a ~10s cold start, go find out
where the server spends it". Two things kill that reading:

- idle does not predict it. Median gap before a slow call is 3 minutes, before a
  fast one 1 minute; 16% of slow calls follow a >=10min gap vs 11% of fast ones.
- the resident server is not where the time goes. It is an outbound HTTP call
  with no deadline, on the hot path, inside a Promise.all.

The waiting also bought nothing. FTS runs in that same Promise.all and is
synchronous, so its rows were already in hand — the request just sat on the
socket holding them.

## Fix

AbortSignal.timeout, default 2500ms, EMBEDDING_TIMEOUT_MS to override. 2.5s cuts
the entire stall cluster (282 of the 286 calls it touches) at the cost of one
legitimately slow embed; tightening to 1s would eat 84 more real ones and gain
nothing, since the cluster is already gone by then.

Resolved per call rather than at module load. Capturing env at import time makes
the knob untunable by anything that configures itself after the import — which
is every embedder of this library, and every test. Same trap as the DB path.

## Degradation stays visible

A timeout returns null, the vector leg is skipped, and RRF proceeds on FTS rows.
That is a quality degradation, so it is labelled rather than swallowed: the
result carries _degradedTo / _degradeReason = 'embedding-timeout', matching what
the early-bail branch already sets, and the timeout logs as its own line instead
of reading like a broken upstream. Fast-and-quietly-worse is the failure mode
this is guarding against.

generateEmbedding is now exported. The timeout is only testable through it —
going via recallMemoriesHybrid cannot discriminate on a machine without the
sqlite-vec extension (CI), because that path early-bails to FTS before it ever
embeds and would pass either way.

## Verification

Red-then-green with the timeout removed: the stalled call runs 30024ms against a
30s stall — it waits the whole thing out — and returns an object instead of null
with no degradation reason. Exactly those three assertions fail; both
healthy-path assertions stay green, so they are not coupled to the fix.

Full suite green: embedding-timeout 7 · cold-pool 6 · recall-contract 14 ·
ranking-importance 5 · recall-endpoint 17 · query-rewrite 6 · memory-health 68 ·
locations 51 · provenance 25 · encoding-damage 18 · injection-hygiene 14 ·
level-migration 10 · supersede-shrink 24 · anchor-pinned 4.

[prediction] 修复: hybrid 的第二个模态(~10.6s 那簇, 占 20%)消失, p90 从 10635ms
  掉到 2500ms 量级; 发起方多数在 1500ms 就放弃了, 那部分白烧的服务端时间一起省掉
[prediction] 风险: 那 20% 的调用改为只有 FTS 结果, 排序质量下降。如果开始出现
  "召回变笨了", 先看 _degradeReason='embedding-timeout' 的比例——比例高说明该治
  上游连通性(代理/DNS), 不是把超时调回去
[prediction] 验证: 观察期后 hybrid 的 p85/p90 应落在 2500ms 附近而不是 10.6s;
  且带 embedding-timeout 标记的调用数应约等于原先 >=3s 的调用数

Co-Authored-By: 千夏 <qianxia@clawgamers.com>
@MXAntian
MXAntian force-pushed the fix/bound-the-embedding-call branch from ce2275b to ed04c25 Compare September 1, 2026 14:54
@MXAntian
MXAntian merged commit c2618d3 into main Sep 1, 2026
2 checks passed
@MXAntian
MXAntian deleted the fix/bound-the-embedding-call branch September 1, 2026 14:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants