improvement(knowledge): scan processing recovery per connector and prune finished outbox events

[waleedlatif1]

Summary

• Processing recovery discovered candidates by walking the global (uploaded_at, id) recovery index, joining each document to its knowledge base and connector, and rejecting it only afterwards. Retained inputs of paused or dormant sources sit at the front of that order, so every call read through all of them first. Follow-up batches (NOT IN attempted connectors) walked the same range again
• Discovery now starts from the eligible connectors and reads each one's oldest candidates through a new partial index doc_connector_processing_recovery_idx (connector_id, uploaded_at, id) with a CROSS JOIN LATERAL, then takes the oldest overall. Paused sources cost nothing. Batch size, the follow-up batches for refused connectors, blocked knowledge bases, the lock-time recheck, and liveness are unchanged, so recovery picks the same candidates as before
• The old doc_processing_recovery_idx stays for now and is dropped in a later contract migration
• Outbox retention: outbox_event never deleted completed rows. The outbox processor now prunes completed rows older than 7 days in bounded, SKIP LOCKED batches, only for knowledge.document.processing.recover and knowledge.document.storage.cleanup. Both have random ids, and nothing reads their completed rows. Every other event type is kept, including idempotency-keyed ones and the checkpoint expiry events, whose completed rows are read back by id, as are pending, processing, and dead-letter rows

Type of Change

• Improvement

Testing

• stored-document-recovery.integration.ts:
  • with 5,000 older paused-source documents ahead of a recoverable one, discovery stays under 1,000 buffers (EXPLAIN ANALYZE BUFFERS) and recovers it in one call. The old query read 10,516 buffers
  • a connector whose full batch is refused doesn't block a sibling's candidate in the same call
• retention.integration.ts: deletes only old completed allowlisted rows and drains past one batch. Removing the status, age, or type filter, or the batch loop, turns it red
• Migration replayed twice (concurrent build, index valid); check:migrations origin/staging passes
• Type-check, lint, check:audits, and the recovery, outbox, and document unit suites pass

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 26, 2026 5:41pm UTC

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[High risk] Database schema and outbox event retention logic.

The PR appears safe to merge; no new actionable issue or outstanding previous finding was identified.

Summary

The PR changes stored-document recovery to page candidates per eligible connector and adds bounded retention for completed recovery and storage-cleanup outbox events.

• Adds a partial per-connector recovery index while retaining the existing index for a later migration.
• Limits each retention run to one batch per allowlisted event type and updates integration coverage.

_{Reviews (2) · Last reviewed commit: "improvement(outbox): prune one bounded b..."}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 11 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

<sub>Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic</sub>