fix(mothership): admit stored content to Chat per item through one gate (#8941)

* test(mothership): record non-Chat responses of shared content routes before gating

Seeds one clean, secret-bearing, unknown and unrecorded item per content store and snapshots
every shared content route as an API-key caller sees it, before any Chat delivery gate changes.

* fix(mothership): admit stored content to Chat per item through one gate

Content from workspace files, run files, knowledge and table rows now reaches
Chat's model only after each item is classified and admitted by the gate of
the surface it leaves through: the registry gate (Chat tools, CLI, mounts) or
the session gate (run_code sandbox proxy). Unknown items are excluded per
item instead of withholding whole results; items with recorded secrets are
redacted where text, refused where opaque. Non-Chat callers are unchanged.

Every operation that admits Copilot declares contentDelivery, and a sourced
use case must account for what it returns, or Chat's transports refuse the
response. Closes the bulk-download, search, run-file, base64, knowledge,
share, export, table and tools-execute paths, and redacts grep lines before
clipping them.

* fix(mothership): address review round on content delivery

Committed writes answer saved-not-shown instead of a retryable error, keep
their panel refresh, and account per operation. Chat streams pin inline
bytes off, taint needs a registry, compile admits before its cache, owned
files bind to their context, and count/truncation oracles are closed for
Chat in knowledge, files search and row search. Table selection cells are
projected before escaping and coerced import cells keep their taint.

* fix(mothership): close sharing, line-window and snapshot gaps in content delivery

Chat may share or see links only to clean files whatever the surface, text
is redacted before a line window cuts it, accounting follows a decision,
cached compiles and knowledge metadata bind to the versions they show,
row-search matches classify at their matched revision, and table counts
follow admitted rows.

* fix(mothership): refuse Chat table predicates over unsafe cells

A table read whose rows, counts, cursor or affected count depend on cell
values is refused for Chat, before the predicate runs, when the table has an
unvouched row or a secret in a referenced column. Inline run-file bytes are
collected per storage key, and table-selection fallbacks are projected.

* fix(mothership): keep content delivery scoped and consistent

* fix(mothership): preserve protection across content transformations

* fix(mothership): preserve file read semantics

* fix(mothership): preserve sharing and search semantics

* fix(mothership): bind media admission to loaded files

* fix(mothership): preserve content revisions and archive budgets

* fix(mothership): preserve partial downloads during compilation

* fix(sandbox): preserve broker error identity

---------

Co-authored-by: Vikhyath Mondreti <vikhyath@simstudio.ai>

Author: waleedlatif1

.claude/rules/sim-content-delivery.md

@@ -0,0 +1,104 @@
+---
+description: How stored content (workspace files, run files, knowledge, table rows) reaches Chat's model, and the per-path policy. Read before changing a use case, route or Chat surface that returns such content.
+paths:
+  - "apps/sim/lib/core/application/**"
+  - "apps/sim/lib/mothership/content-delivery/**"
+  - "apps/sim/lib/workspace-files/application/**"
+  - "apps/sim/lib/knowledge/application/**"
+  - "apps/sim/lib/table/application/**"
+  - "apps/sim/lib/workflows/application/**"
+  - "apps/sim/lib/mothership/chat/application/**"
+  - "apps/sim/lib/**/application/operations.ts"
+  - "apps/sim/lib/mothership/agent-cli/**"
+  - "apps/sim/lib/mothership/tools/**"
+  - "apps/sim/app/api/v2/**"
+---
+
+# Sim Content Delivery to Chat
+
+Content derived from a provenance-bearing store reaches Chat's model only after **each item** it
+came from is admitted by the gate of the surface it leaves through. The stores are workspace files
+(and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers
+bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output,
+error text that quotes content, and public links. Counts, ordering, and match locations are ordinary
+metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional
+human uploads remain ordinary input and are not scanned.
+
+Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are
+classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and
+the workflow executor see byte-identical results.
+
+## The pieces
+
+- `lib/core/application/content-delivery.ts`: the primitive.
+  - `admitContent(principal, sources)` returns per-item verdicts. `requireAdmittedContent` is the
+    single-item form and throws `ContentRefusedError` (403).
+  - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure
+    does not undo the write: audit and effects run, then the operation answers with the refusal.
+    Settled tool results use the same deferral, but do not claim that a mutation was saved.
+  - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry
+    may appear), `{ derived: value }` (only entries occurring in `value`), `'opaque'` (bytes no
+    redaction reaches: binary, base64, archives, images, public links), or `'excerpt'` (text already
+    cut before admission, where a secret fragment cannot be redacted). Both gates withhold an
+    excerpt with known secret entries; raw opaque bytes may still enter the workbench with history.
+  - `redactBeforeCutting` projects complete text before selecting a line window. The session gate
+    creates the existing redactor from the operation's bounded evidence only when a cut needs it;
+    ordinary byte reads do not decrypt a second time or reload history.
+- Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries
+  into the registry egress projects with; the **session gate** (run_code sandbox proxy) records
+  them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct
+  single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's
+  registry so the whole output is withheld and the workbench copy stays unknown. It needs a
+  registry to latch; without one it refuses. One refused item never latches the shared registry.
+- `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches
+  the model (the user's own edit preview; a derived file that carries its sources' provenance).
+  Name why at each use.
+
+## Rules
+
+1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result
+   can carry content from a store, `'none'` otherwise. Definition throws
+   without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one.
+2. **A `sourced` use case accounts for what it returns**, before it loads content: call
+   `admitContent` (with `[]` when this call returns none). Otherwise Chat gets
+   `ContentDeliveryUnaccountedError`, and the transports turn the response into a 403.
+3. **Classify before loading bytes**, and exclude refused items per item. Withhold a whole result
+   only when its items cannot be separated (one file, one render, one compile).
+4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.**
+   `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive.
+   Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`.
+5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for
+   Chat goes through the gated use case, or names `observeContentOutsideModel` and why.
+
+## Policy
+
+U = `unknown`, E = exact with secret entries, R/C = unrecorded or clean. Each item is decided on its
+own unless noted.
+
+| Path | U | E | R / C |
+|---|---|---|---|
+| File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered |
+| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered |
+| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered |
+| File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered |
+| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included |
+| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned |
+| File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
+| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
+| Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
+| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references |
+| Run file download | 403 | Redacted | Delivered |
+| Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) |
+| KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered |
+| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source |
+| Table export download | Existing unsupported Chat operation | | |
+| Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
+| Row search | Locations returned without cell values | Same | Returned |
+| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
+| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization |
+
+Search pagination and aggregate totals retain their database meaning even when an item or preview is
+withheld. Do not scan unrelated table rows or whole document collections to conceal counts.
+A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path.
+Registry admission commits only when the complete item fits the existing capacity; a refused item
+must not consume capacity or permanently disable the parent registry.

.cursor/rules/sim-content-delivery.mdc

@@ -0,0 +1,95 @@
+---
+description: "How stored content (workspace files, run files, knowledge, table rows) reaches Chat's model, and the per-path policy. Read before changing a use case, route or Chat surface that returns such content."
+globs: ["apps/sim/lib/core/application/**","apps/sim/lib/mothership/content-delivery/**","apps/sim/lib/workspace-files/application/**","apps/sim/lib/knowledge/application/**","apps/sim/lib/table/application/**","apps/sim/lib/workflows/application/**","apps/sim/lib/mothership/chat/application/**","apps/sim/lib/**/application/operations.ts","apps/sim/lib/mothership/agent-cli/**","apps/sim/lib/mothership/tools/**","apps/sim/app/api/v2/**"]
+---
+
+<!-- Generated from .claude/rules/sim-content-delivery.md by `bun run skills:sync`. Edit the source, not this file. -->
+
+# Sim Content Delivery to Chat
+
+Content derived from a provenance-bearing store reaches Chat's model only after **each item** it
+came from is admitted by the gate of the surface it leaves through. The stores are workspace files
+(and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers
+bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output,
+error text that quotes content, and public links. Counts, ordering, and match locations are ordinary
+metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional
+human uploads remain ordinary input and are not scanned.
+
+Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are
+classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and
+the workflow executor see byte-identical results.
+
+## The pieces
+
+- `lib/core/application/content-delivery.ts`: the primitive.
+  - `admitContent(principal, sources)` returns per-item verdicts. `requireAdmittedContent` is the
+    single-item form and throws `ContentRefusedError` (403).
+  - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure
+    does not undo the write: audit and effects run, then the operation answers with the refusal.
+    Settled tool results use the same deferral, but do not claim that a mutation was saved.
+  - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry
+    may appear), `{ derived: value }` (only entries occurring in `value`), `'opaque'` (bytes no
+    redaction reaches: binary, base64, archives, images, public links), or `'excerpt'` (text already
+    cut before admission, where a secret fragment cannot be redacted). Both gates withhold an
+    excerpt with known secret entries; raw opaque bytes may still enter the workbench with history.
+  - `redactBeforeCutting` projects complete text before selecting a line window. The session gate
+    creates the existing redactor from the operation's bounded evidence only when a cut needs it;
+    ordinary byte reads do not decrypt a second time or reload history.
+- Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries
+  into the registry egress projects with; the **session gate** (run_code sandbox proxy) records
+  them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct
+  single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's
+  registry so the whole output is withheld and the workbench copy stays unknown. It needs a
+  registry to latch; without one it refuses. One refused item never latches the shared registry.
+- `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches
+  the model (the user's own edit preview; a derived file that carries its sources' provenance).
+  Name why at each use.
+
+## Rules
+
+1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result
+   can carry content from a store, `'none'` otherwise. Definition throws
+   without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one.
+2. **A `sourced` use case accounts for what it returns**, before it loads content: call
+   `admitContent` (with `[]` when this call returns none). Otherwise Chat gets
+   `ContentDeliveryUnaccountedError`, and the transports turn the response into a 403.
+3. **Classify before loading bytes**, and exclude refused items per item. Withhold a whole result
+   only when its items cannot be separated (one file, one render, one compile).
+4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.**
+   `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive.
+   Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`.
+5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for
+   Chat goes through the gated use case, or names `observeContentOutsideModel` and why.
+
+## Policy
+
+U = `unknown`, E = exact with secret entries, R/C = unrecorded or clean. Each item is decided on its
+own unless noted.
+
+| Path | U | E | R / C |
+|---|---|---|---|
+| File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered |
+| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered |
+| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered |
+| File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered |
+| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included |
+| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned |
+| File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
+| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
+| Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
+| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references |
+| Run file download | 403 | Redacted | Delivered |
+| Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) |
+| KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered |
+| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source |
+| Table export download | Existing unsupported Chat operation | | |
+| Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
+| Row search | Locations returned without cell values | Same | Returned |
+| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
+| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization |
+
+Search pagination and aggregate totals retain their database meaning even when an item or preview is
+withheld. Do not scan unrelated table rows or whole document collections to conceal counts.
+A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path.
+Registry admission commits only when the complete item fits the existing capacity; a refused item
+must not consume capacity or permanently disable the parent registry.

apps/sim/AGENTS.md

@@ -9,6 +9,7 @@ Applies to `apps/sim/**` on top of the root [AGENTS.md](/AGENTS.md), which holds
 - Styling and emcn chrome, UI copy, settings pages: `sim-styling.md`, `sim-ui-copy.md`, `sim-settings-pages.md`
 - URL state, list and menu order, caching: `sim-url-state.md`, `sim-list-ordering.md`, `sim-caching.md`
 - Tools, blocks, triggers: `sim-integrations.md`; isolated-vm sandbox worker: `sim-sandbox.md`
+- Anything that returns stored file, run-file, knowledge or table content to Chat: `sim-content-delivery.md`
 - Tests: `sim-testing.md` and the `test-audit` skill
 - Landing pages: `app/(landing)/CLAUDE.md`, `landing-seo-geo.md`, `constitution.md` (product language)
 

apps/sim/app/api/v2/files/bulk-download/route.test.ts

@@ -76,6 +76,7 @@ describe('GET /api/v2/files/bulk-download', () => {
       folderPaths: new Map<string, string>(),
       renderedDocuments: new Map<string, Buffer>(),
       declaredBytes: 6,
+      excludedFileIds: [],
     })
   })
 

apps/sim/app/api/v2/files/bulk-download/route.ts

@@ -54,14 +54,16 @@ export const GET = defineV2BinaryRoute({
     folderPaths: query.folderPaths,
   }),
   useCase: downloadWorkspaceFileItems,
-  present: ({ filesToZip, folderPaths, renderedDocuments }) => {
-    const entryPaths = buildZipEntryPaths(
-      filesToZip.map((file) => ({
+  present: ({ filesToZip, folderPaths, renderedDocuments, excludedFileIds }) => {
+    const report = excludedFileIds.length > 0 ? { partial: true, excludedFileIds } : undefined
+    const entryPaths = buildZipEntryPaths([
+      ...filesToZip.map((file) => ({
         name: file.name,
         folderPath: file.folderId ? folderPaths.get(file.folderId) : null,
         contentType: file.type,
-      }))
-    )
+      })),
+      ...(report ? [{ name: 'sim-download-report.json', contentType: 'application/json' }] : []),
+    ])
     const archive = new ZipArchive({ store: true })
     archive.on('warning', (error: Error) => {
       logger.warn('Archive warning while streaming workspace files', { error })
@@ -71,6 +73,9 @@ export const GET = defineV2BinaryRoute({
         name: entryPaths[index],
       })
     })
+    if (report) {
+      archive.append(JSON.stringify(report, null, 2), { name: entryPaths[filesToZip.length] })
+    }
     archive.finalize().catch((error) => {
       logger.error('Failed to finalize workspace file archive', { error })
     })

apps/sim/app/api/v2/lib/response.ts

@@ -10,6 +10,7 @@ import { type CursorKey, INVALID_CURSOR_MESSAGE } from '@/lib/api/list-query'
 import { getValidationErrorMessage, serializeZodIssues } from '@/lib/api/server'
 import { ADMISSION_RETRY_AFTER_SECONDS } from '@/lib/core/admission/transient-failure'
 import {
+  ContentDeliveryUnobservedError,
   forbiddenErrorDetails,
   InsufficientScopeError,
   OAuthAccessTokenExpiredError,
@@ -530,6 +531,10 @@ export function v2ErrorForOrchestration(
  * one its failure class maps to.
  */
 export function v2CaughtOrchestrationError(error: unknown): NextResponse | null {
+  /** Chat reached stored content outside any classifying surface: transient, never a content leak. */
+  if (error instanceof ContentDeliveryUnobservedError) {
+    return v2Error('SERVICE_UNAVAILABLE', error.message)
+  }
   const classified = asOrchestrationError(error)
   if (!classified) return null
   if (classified instanceof InsufficientScopeError) return v2InsufficientScope(classified)

apps/sim/ee/access-requests/lib/application/operations.ts

@@ -51,6 +51,7 @@ function defineAccessRequestOperation(
          * permission-group-exempt: requests never grant a withheld capability without administrator review.
          */
         capability: 'none',
+        contentDelivery: 'none',
       })
   return Object.freeze({ ...workspaceOperation, organizationOperation, admin })
 }