fix(mothership): make chat fork complete, consistent and safe to retry

[waleedlatif1]

Summary

Sim side of making chat fork work for long chats and leave nothing inconsistent behind. Companion worker PR: simstudioai/mothership#579 (set-based copy, 50,000-event ceiling with 413, rollback when the caller disconnects).

• Typed refusals. copyWorkerConversation turned every worker non-2xx into a generic error, so the route answered 500 even when the worker refused for a reason the person can act on. Worker 404 → not_found, 409 → conflict, 413 → payload_too_large; the fork route passes 409/413 through with a caller-facing message (404/400 mappings unchanged), and the fork action's toast shows that message instead of "Failed to fork chat". The route's existing 413 for more than 200 inline images now also reaches the person instead of becoming a 500.
• No overlapping retries. The retry loop's own catch swallowed every first-attempt failure, so a 400, a 500, a malformed receipt and a timed-out attempt were all resent while the first copy could still be running. Only a recognized socket failure (isRetryableNetworkError, which now includes EHOSTUNREACH: a connection that never opened) or a 502/503/504 gets one more attempt.
• Per-attempt timeout 15 s → 120 s, sized from the worker's measured copy time at its ceiling (table below) with headroom for a slower production database (50,000 events take 5–9 s locally; an estimated up-to-~8x production factor from one sample puts that at 40–70 s). App and worker load balancers keep idle connections for an hour, so nothing in between cuts the attempt. A legitimate fork finishes inside one attempt; one that doesn't is abandoned and never retried, and the worker rolls its copy back when the connection closes.
• Compensation. If anything after the worker copy fails (most often the final transaction that publishes the chat), the fork now asks the worker to clean the new chat up through the existing POST /api/tasks/cleanup, the endpoint chat deletion already uses. Best effort, logged, and a no-op when nothing was committed.
• References point at what the fork holds:
  • A file whose blob copy failed is never published, yet messages and the worker's maps were rewritten to its id/key. Failed copies are now left out of the maps, so references stay on the source file (alive while the source is); its resource tab is dropped as before and failedFileCopies still reports it.
  • In-app /workspace/<ws>/files/<id> links are rewritten (the fork stays in the same workspace, so only the file id moves).
  • Tool-call params (a string that is a mapped id or key is replaced whole; other strings get the URL grammar), activityDescription and display.title are rewritten.
  • A Sources tab whose response is past the cut (matched by message id or request id, as the panel does) is no longer copied.
• Fork vs. purge ordering. A fork can share keys with its source (organization attachments, files whose copy failed), and cleanup deletes a shared key only once no remaining chat references it, checking after it deletes the source row. The fork's publish transaction holds the source row (FOR KEY SHARE) until commit, so a purge either already removed it (the fork returns 404 and its worker copy is discarded) or waits for the fork and then sees its references.
• Purge:
  • Chat images (chat-images/<chatId>/…) were never deleted with their chat. Their keys are rebuilt from the assistant messages that published them, sharing the fork's enumeration (moved to inline-image-key.ts so cleanup doesn't load sharp).
  • Message attachments were deleted as copilot storage whatever their key. Chat attachments are workspace-bucket files owned by their workspace_files row, a fork can share one with its source (failed copy, soft-deleted file), and the copilot bucket falls back to the workspace bucket on GCS (GCS_COPILOT_BUCKET_NAME unset) and can be configured to it on S3. So purging either chat could delete a file the other, or the workspace, still uses. Workspace keys are now left to their rows; copilot/ keys are deleted as before.
  • Organization attachments (assistant/<orgId>/…) have no workspace_files row and a fork carries the same key. They are deleted under their own context, and only once no remaining chat of that organization references the key (checked after the purged rows are gone, so a fork or a soft-deleted chat keeps it).

Deferred (need an owner decision)

• autoAllowedTools is not carried to the fork. It is a per-chat consent grant ("allow for the rest of this chat"). Copying it widens consent to a chat the person never granted it in; not copying it re-prompts. That is a product/security call, not a bug fix.
• A forked task_armed pill. Pill state comes from the transcript, and the source's background task notifies only the source, so a fork cut between arm and notification shows the task pending forever. Whether a fork should show it as not carried over, mirror the source's status, or re-arm its own task is a product decision; the worker PR defers it too.
• Legacy copilot/ attachment keys shared by a source and its fork. Chat uploads never produce them (they are workspace/ or assistant/ keys), so this only affects old rows; closing it needs a reference check across chats or copying those blobs on fork.

Mixed versions and deploy order

Worker first is preferred; every combination is safe:

• New worker, old Sim: old Sim retries any failure once with a 15 s timeout. Large forks that take longer time out, the worker rolls the copy back on disconnect, and the retry either finds the same receipt (destination lock + request hash) or is rolled back too. No orphan, but large forks still fail until Sim deploys.
• Old worker, new Sim: the old worker never returns 413 and doesn't roll back on disconnect; it also can't copy past ~13k events (bind limit), so those still fail, now as one attempt. A timed-out attempt on the old worker can commit after Sim's cleanup call, leaving an orphan worker conversation; that window closes when the worker deploys.
• Both new: the remaining window is a worker that passes its last disconnect check and commits in the same instant Sim gives up; documented on discardWorkerConversation.
• No schema, contract or migration changes; the route's success response is unchanged.

Worker copy time (local, relative)

From the worker PR: prod-shaped synthetic chats, cut at the last turn with its response, 5 reps, a separate writer process appending to the source, local Postgres 18 on a loaded machine. Use for ratios.

events / MB	before p50 / p95	after p50 / p95	peak RSS before → after
1,630 / 10	0.48 / 0.50 s	0.31 / 0.42 s	184 → 78 MB
8,281 / 30	1.33 / 2.56 s	0.85 / 0.96 s	491 → 110 MB
15,348 / 63	fails (bind limit)	1.46 / 2.26 s	860 → 117 MB
50,000 / 204	fails	4.7 / 5.4 s	2.6 GB → 216 MB
99,900 / 405	fails	10.6 / 13.3 s (measured above the ceiling; ships as 413)	5.0 GB → 235 MB

Type of Change

• Bug fix

Testing

• Red on origin/staging, green now (each guard reverted on its own and its test watched go red):
  • fork-worker.test.ts (fake worker recording the requests it receives): 404/409/413 classified with one request; 500/400, a timed-out attempt and a non-socket TypeError not repeated; EHOSTUNREACH/ENETUNREACH/ECONNRESET and 502/503/504 retried once; a 404 whose body fails to cancel still classified.
  • fork/route.test.ts: worker 404/409/413 passed through; a failed publish sends the worker a cleanup for the new chat id (red without the discard); a failed blob copy leaves the message and the worker maps on the source file (red with the full plan maps); in-app links and tool-call arguments rewritten (red without the workspace id, and red without the whole-value match); a Sources tab past the cut dropped and one addressed by a kept request id kept (red without the filter, and red without the request-id match).
  • fork/route.test.ts: a source purged while the fork copied is refused with 404, nothing published, worker copy discarded (red without the source-row check).
  • chat-cleanup.test.ts through prepareChatCleanup: a workspace attachment never deleted as copilot storage (red without the prefix guard); an organization attachment deleted only when no remaining chat references it (red without the reference check, and red without collecting it); a chat's inline images deleted (red without the collection).
• Sim end-to-end (scratch copy of the Chat E2E harness; real local Sim app over HTTP, real Postgres/Redis, two worker processes from the companion worker branch on Bun 1.4.2, scripted model stub): a two-turn chat with an upload, forked at the first answer through POST /api/mothership/chats/[chatId]/fork → the upload is copied (new key, bytes on disk), Sim's and the worker's fork history name only the copy, the worker receipt is at cut+1, and a new turn on the fork completes with the model seeing the kept turn and not the dropped one. An unknown message id is refused with 400. JSON report written per check.
• vitest run lib/mothership app/api/mothership lib/cleanup lib/core/errors background/cleanup: 2,813 pass.
• bun run test from the root, bun run lint, apps/sim type-check, bun run check:audits (58), check:api-validation:strict, docs-manifest:check all pass.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing (new tests pass the test-audit authoring gate)
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Oct 2, 2026 8:02pm UTC

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 15 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[greptile-apps]

Confidence Score: 5/5

[High risk] Refactors chat fork operation to handle edge cases and worker failures.

The PR appears safe to merge; no outstanding finding or actionable new issue was established.

Summary

The PR makes chat forks safer to retry and clean up, preserves file references when copies fail, and improves purge behavior. Changes since the previous review also add paginated Lucid and Notion browsing and bounded Notion date searches.

• Fork publication now checks and locks the source chat before committing the fork.
• Live search adds browse modes, signed continuations, provider access checks, and discovery tests.

_{Reviews (3) · Last reviewed commit: "chore(mothership): quote the final copy ..."}

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 15 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 15 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}