Skip to content

fix(extraction): retry transient model failures once and give a failed terminal flush one retry - #997

Draft
gorkem2020 wants to merge 1 commit into
CortexReach:masterfrom
gorkem2020:fix/transient-extraction-resilience
Draft

fix(extraction): retry transient model failures once and give a failed terminal flush one retry#997
gorkem2020 wants to merge 1 commit into
CortexReach:masterfrom
gorkem2020:fix/transient-extraction-resilience

Conversation

@gorkem2020

Copy link
Copy Markdown
Contributor

Summary

Two deferred items from earlier review rounds (#927 and #934), both about the extraction path's behavior when the model or the gateway does not answer.

  1. completeJson folds every failure into null, so the extraction path could not tell a silent upstream (timeout, 5xx, connection reset, host-transport failure) from an unusable answer. The grounding rejudge failed closed on both, demoting every real-tagged durable in the batch, and the batch counted as settled. Failing closed on a verdict is right; failing closed on silence should not look the same.
  2. When a terminal flush's extraction failed, the consumed turns were handed back to the deferred-flush bucket, but the session had already ended and nothing consumed that bucket again. The existing regression only passed because it emitted a second session_end by hand.

Changes

  • src/extraction-transient-retry.ts (new): completeJsonWithTransientRetry wraps a model call, reads the client's last error after a null, and classifies it with the classifier the repository already ships in src/reflection-retry.ts. An upstream request failure of the transient class is retried once after the shared backoff; an upstream failure of either class reports the run as unavailable; a parse failure or empty answer is still treated as the model's answer (no retry, not unavailable).
  • src/smart-extractor.ts: both model calls in the extraction path (extract-candidates, grounding-rejudge) go through that wrapper. When the judge is unavailable the batch is handed back (status: "llm_unavailable") instead of failing closed; nothing is persisted and nothing is demoted. SmartExtractorConfig.transientRetrySleep is a test seam for the backoff.
  • src/memory-categories.ts: ExtractionStats.llmUnavailable, always paired with extractionFailed, so the caller can tell "the model never answered" from "the model answered nothing usable".
  • index.ts: the auto-capture hook logs the unavailable case explicitly (the rate limiter was already not charged on extractionFailed). A terminal flush whose extraction fails now schedules exactly one unref()ed retry per session key (15 s); the retry re-runs the terminal flush for that key, is cancelled when any later run consumes the bucket, and gives up after its single attempt. _setAutoCaptureTerminalFlushRetryDelayMsForTest shortens the delay for tests only.
  • Tests: test/extraction-transient-retry.test.mjs (upstream-failure detection, retry-once outcomes for transient, non-retryable, unusable-answer and no-diagnostics clients, and a full extractAndPersist run whose judge never answers: deferred, nothing persisted, nothing demoted) and test/autocapture-terminal-flush-retry.test.mjs (a failed terminal flush retries from its own timer and persists on the retry; a second failure gives up without a third attempt; a later flush that consumes the bucket cancels the pending retry). Both registered in the npm test chain and the CI manifest (core-regression).

Notes

  • The API-key client already absorbs most single transients inside the OpenAI SDK; the plugin-level retry mainly covers the host transport and the OAuth path, and it uses the same delay and classifier as the reflection lane so there is one transient policy in the repository.
  • No cursor semantics change on the non-terminal path: a deferred batch is restored the same way a failed extraction already was, and the rate limiter is not charged, so a later turn re-runs the extraction on the same input.

Verification

  • npm run build, npm test, the two new regression files (red against the previous code by construction), and the existing grounding-rejudge, extraction-grounding-register, autocapture-watermark-reset, autocapture-fallback-gating, smart-extractor-noise-gating, reflection-embed-transient-retry and context-support-e2e suites.

…d terminal flush one retry

completeJson folds every failure into null, so the extraction path could
not tell a silent upstream (timeout, 5xx, connection reset, host-transport
failure) from an unusable answer; the grounding rejudge failed closed on
both and demoted every real-tagged durable in the batch, and the batch
counted as settled. A terminal flush whose extraction failed handed its
turns back to the deferred bucket, but the session had already ended and
nothing consumed that bucket again.

Both model calls in the extraction path now go through
completeJsonWithTransientRetry: the client's last error classifies a null
with the repository's existing transient classifier, a transient upstream
failure is retried once after the shared backoff, and an upstream failure
of either class reports the run as unavailable (ExtractionStats.llmUnavailable,
paired with extractionFailed) so the caller defers the batch instead of
judging it. A terminal flush whose extraction fails schedules exactly one
unref()ed retry per session key; the retry re-runs the terminal flush, is
cancelled when a later run consumes the bucket, and gives up after its
single attempt.

Regressions: upstream-failure detection, retry-once outcomes for the
transient, non-retryable, unusable-answer and no-diagnostics clients, a
full extractAndPersist run whose judge never answers (deferred, nothing
persisted, nothing demoted), a failed terminal flush retried from its own
timer, a second failure that gives up, and a later flush cancelling the
pending retry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant