fix(mothership): keep content delivery scoped and consistent

Author: icecrasher321

.claude/rules/sim-content-delivery.md

@@ -20,7 +20,9 @@ Content derived from a provenance-bearing store reaches Chat's model only after
 came from is admitted by the gate of the surface it leaves through. The stores are workspace files
 (and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers
 bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output,
-error text that quotes content, public links, and match results.
+error text that quotes content, and public links. Counts, ordering, and match locations are ordinary
+metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional
+human uploads remain ordinary input and are not scanned.
 
 Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are
 classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and
@@ -33,6 +35,7 @@ the workflow executor see byte-identical results.
     single-item form and throws `ContentRefusedError` (403).
   - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure
     does not undo the write: audit and effects run, then the operation answers with the refusal.
+    Settled tool results use the same deferral, but do not claim that a mutation was saved.
   - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry
     may appear), `{ derived: value }` (only entries occurring in `value`), or `'opaque'` (bytes no
     redaction reaches: binary, base64, archives, images, public links).
@@ -49,7 +52,7 @@ the workflow executor see byte-identical results.
 ## Rules
 
 1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result
-   can carry content from a store (including a match oracle), `'none'` otherwise. Definition throws
+   can carry content from a store, `'none'` otherwise. Definition throws
    without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one.
 2. **A `sourced` use case accounts for what it returns**, before it loads content: call
    `admitContent` (with `[]` when this call returns none). Otherwise Chat gets
@@ -58,7 +61,7 @@ the workflow executor see byte-identical results.
    only when its items cannot be separated (one file, one render, one compile).
 4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.**
    `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive.
-   Refresh the list with `vitest run -u lib/api/server/routes/copilot-route-inventory.test.ts`.
+   Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`.
 5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for
    Chat goes through the gated use case, or names `observeContentOutsideModel` and why.
 
@@ -70,19 +73,27 @@ own unless noted.
 | Path | U | E | R / C |
 |---|---|---|---|
 | File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered |
-| File download (run_code proxy) and mounts | Refused, not recorded, listed in `excludedInputs` | Recorded in workbench history | Delivered |
+| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered |
+| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered |
 | File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered |
-| Bulk download zip | Left out | Left out (opaque) | Included |
-| `files search` | Results dropped | Results dropped (a match is an oracle) | Returned |
+| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included |
+| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned |
 | File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
 | Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
 | Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
-| Doc compile | Compile refused | Error text redacted | Compiles |
+| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references |
 | Run file download | 403 | Redacted | Delivered |
 | Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) |
 | KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered |
-| KB export, table export download | Not available to Chat | | |
+| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source |
+| Table export download | Existing unsupported Chat operation | | |
 | Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
-| Row search | Match dropped | Match in a secret column dropped | Returned |
+| Row search | Locations returned without cell values | Same | Returned |
 | Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
-| `tools execute` for file, knowledge, table, memory, function, custom, MCP tools | Not available to Chat | | |
+| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization |
+
+Search pagination and aggregate totals retain their database meaning even when an item or preview is
+withheld. Do not scan unrelated table rows or whole document collections to conceal counts.
+A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path.
+Registry admission commits only when the complete item fits the existing capacity; a refused item
+must not consume capacity or permanently disable the parent registry.

.cursor/rules/sim-content-delivery.mdc

@@ -11,7 +11,9 @@ Content derived from a provenance-bearing store reaches Chat's model only after
 came from is admitted by the gate of the surface it leaves through. The stores are workspace files
 (and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers
 bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output,
-error text that quotes content, public links, and match results.
+error text that quotes content, and public links. Counts, ordering, and match locations are ordinary
+metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional
+human uploads remain ordinary input and are not scanned.
 
 Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are
 classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and
@@ -24,6 +26,7 @@ the workflow executor see byte-identical results.
     single-item form and throws `ContentRefusedError` (403).
   - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure
     does not undo the write: audit and effects run, then the operation answers with the refusal.
+    Settled tool results use the same deferral, but do not claim that a mutation was saved.
   - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry
     may appear), `{ derived: value }` (only entries occurring in `value`), or `'opaque'` (bytes no
     redaction reaches: binary, base64, archives, images, public links).
@@ -40,7 +43,7 @@ the workflow executor see byte-identical results.
 ## Rules
 
 1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result
-   can carry content from a store (including a match oracle), `'none'` otherwise. Definition throws
+   can carry content from a store, `'none'` otherwise. Definition throws
    without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one.
 2. **A `sourced` use case accounts for what it returns**, before it loads content: call
    `admitContent` (with `[]` when this call returns none). Otherwise Chat gets
@@ -49,7 +52,7 @@ the workflow executor see byte-identical results.
    only when its items cannot be separated (one file, one render, one compile).
 4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.**
    `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive.
-   Refresh the list with `vitest run -u lib/api/server/routes/copilot-route-inventory.test.ts`.
+   Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`.
 5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for
    Chat goes through the gated use case, or names `observeContentOutsideModel` and why.
 
@@ -61,19 +64,27 @@ own unless noted.
 | Path | U | E | R / C |
 |---|---|---|---|
 | File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered |
-| File download (run_code proxy) and mounts | Refused, not recorded, listed in `excludedInputs` | Recorded in workbench history | Delivered |
+| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered |
+| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered |
 | File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered |
-| Bulk download zip | Left out | Left out (opaque) | Included |
-| `files search` | Results dropped | Results dropped (a match is an oracle) | Returned |
+| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included |
+| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned |
 | File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
 | Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
 | Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
-| Doc compile | Compile refused | Error text redacted | Compiles |
+| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references |
 | Run file download | 403 | Redacted | Delivered |
 | Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) |
 | KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered |
-| KB export, table export download | Not available to Chat | | |
+| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source |
+| Table export download | Existing unsupported Chat operation | | |
 | Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
-| Row search | Match dropped | Match in a secret column dropped | Returned |
+| Row search | Locations returned without cell values | Same | Returned |
 | Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
-| `tools execute` for file, knowledge, table, memory, function, custom, MCP tools | Not available to Chat | | |
+| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization |
+
+Search pagination and aggregate totals retain their database meaning even when an item or preview is
+withheld. Do not scan unrelated table rows or whole document collections to conceal counts.
+A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path.
+Registry admission commits only when the complete item fits the existing capacity; a refused item
+must not consume capacity or permanently disable the parent registry.

apps/sim/app/api/v2/files/bulk-download/route.test.ts

@@ -76,6 +76,7 @@ describe('GET /api/v2/files/bulk-download', () => {
       folderPaths: new Map<string, string>(),
       renderedDocuments: new Map<string, Buffer>(),
       declaredBytes: 6,
+      excludedFileIds: [],
     })
   })
 

apps/sim/app/api/v2/files/bulk-download/route.ts

@@ -54,14 +54,16 @@ export const GET = defineV2BinaryRoute({
     folderPaths: query.folderPaths,
   }),
   useCase: downloadWorkspaceFileItems,
-  present: ({ filesToZip, folderPaths, renderedDocuments }) => {
-    const entryPaths = buildZipEntryPaths(
-      filesToZip.map((file) => ({
+  present: ({ filesToZip, folderPaths, renderedDocuments, excludedFileIds }) => {
+    const report = excludedFileIds.length > 0 ? { partial: true, excludedFileIds } : undefined
+    const entryPaths = buildZipEntryPaths([
+      ...filesToZip.map((file) => ({
         name: file.name,
         folderPath: file.folderId ? folderPaths.get(file.folderId) : null,
         contentType: file.type,
-      }))
-    )
+      })),
+      ...(report ? [{ name: 'sim-download-report.json', contentType: 'application/json' }] : []),
+    ])
     const archive = new ZipArchive({ store: true })
     archive.on('warning', (error: Error) => {
       logger.warn('Archive warning while streaming workspace files', { error })
@@ -71,6 +73,9 @@ export const GET = defineV2BinaryRoute({
         name: entryPaths[index],
       })
     })
+    if (report) {
+      archive.append(JSON.stringify(report, null, 2), { name: entryPaths[filesToZip.length] })
+    }
     archive.finalize().catch((error) => {
       logger.error('Failed to finalize workspace file archive', { error })
     })

apps/sim/executor/utils/resolved-secret-trace-registry.ts

@@ -1731,6 +1731,41 @@ export class ResolvedSecretTraceRegistry {
     return importedAll
   }
 
+  /** Imports one item's trusted evidence atomically, leaving this registry unchanged on refusal. */
+  async tryImportProvenance(
+    provenances: Iterable<ResolvedSecretTraceProvenanceV1>,
+    options: ImportResolvedSecretTraceProvenanceOptions & { value?: unknown }
+  ): Promise<boolean> {
+    const staged = new ResolvedSecretTraceRegistry([], this.scope, { staged: true })
+    for (const provenance of provenances) {
+      if (!(await staged.importProvenance(provenance, options)) || !staged.isComplete())
+        return false
+    }
+
+    let entries = [...staged.activeEntries.values()]
+    if (options.value !== undefined) {
+      const filtered = staged.exportProvenanceForValue(options.value)
+      if (!filtered.complete) return false
+      const selected = new Set(
+        filtered.entries.map((entry) =>
+          activeEntryKey({
+            name: entry.name ?? '',
+            plaintext: '',
+            encryptedValue: entry.encryptedValue,
+            anonymous: entry.name === undefined,
+          })
+        )
+      )
+      entries = entries.filter((entry) => selected.has(activeEntryKey(entry)))
+    }
+
+    /** No await separates the capacity check from activation, including across concurrent calls. */
+    if (this.isPermanentlyIncomplete() || this.activeEntryBytesAfter(entries) === undefined)
+      return false
+    for (const entry of entries) this.addActiveEntry(entry, { propagated: true })
+    return true
+  }
+
   /**
    * Imports only provenance whose decrypted plaintext occurs in the exact value crossing this
    * boundary. Staging in a scoped temporary registry prevents a valid but overbroad envelope from
@@ -2578,6 +2613,22 @@ export class ResolvedSecretTraceRegistry {
     if (copied) target.inheritIncompletenessReasonsFrom(this)
   }
 
+  private activeEntryBytesAfter(entries: Iterable<ActiveSecretEntry>): number | undefined {
+    let count = this.activeEntries.size
+    let bytes = this.activeProvenanceEntryBytes
+    for (const entry of entries) {
+      if (this.activeEntries.has(activeEntryKey(entry))) continue
+      bytes += (count === 0 ? 0 : 1) + serializedProvenanceEntryByteSize(toProvenanceEntry(entry))
+      count += 1
+      if (
+        count > MAX_PROVENANCE_ENTRIES ||
+        this.completeProvenanceEnvelopeBytes + bytes > MAX_SERIALIZED_PROVENANCE_BYTES
+      )
+        return undefined
+    }
+    return bytes
+  }
+
   private addActiveEntry(entry: ActiveSecretEntry, options: { propagated?: boolean } = {}): void {
     const key = activeEntryKey(entry)
     const existing = this.activeEntries.get(key)
@@ -2596,22 +2647,14 @@ export class ResolvedSecretTraceRegistry {
       return
     }
 
-    const entryBytes = serializedProvenanceEntryByteSize(toProvenanceEntry(entry))
-    const separatorBytes = this.activeEntries.size === 0 ? 0 : 1
-    if (
-      this.activeEntries.size >= MAX_PROVENANCE_ENTRIES ||
-      this.completeProvenanceEnvelopeBytes +
-        this.activeProvenanceEntryBytes +
-        separatorBytes +
-        entryBytes >
-        MAX_SERIALIZED_PROVENANCE_BYTES
-    ) {
+    const nextEntryBytes = this.activeEntryBytesAfter([entry])
+    if (nextEntryBytes === undefined) {
       this.markIncomplete('provenance-capacity-exceeded')
       return
     }
     this.activeEntries.set(key, entry)
     if (options.propagated) this.propagatedEntryKeys.add(key)
-    this.activeProvenanceEntryBytes += separatorBytes + entryBytes
+    this.activeProvenanceEntryBytes = nextEntryBytes
     this.modelEgressRevision += 1
   }
 

apps/sim/lib/api/server/routes/copilot-content-routes.json

@@ -86,8 +86,8 @@
   },
   {
     "method": "GET",
-    "path": "/api/v2/knowledge/[knowledgeBaseId]/tags/usage",
-    "operation": "knowledge.tags.read_usage"
+    "path": "/api/v2/knowledge/[knowledgeBaseId]/export",
+    "operation": "knowledge.export"
   },
   {
     "method": "POST",
@@ -131,13 +131,13 @@
   },
   {
     "method": "POST",
-    "path": "/api/v2/tables/[tableId]/rows/search",
-    "operation": "tables.rows.search"
+    "path": "/api/v2/tables/[tableId]/rows/upsert",
+    "operation": "tables.rows.upsert"
   },
   {
     "method": "POST",
-    "path": "/api/v2/tables/[tableId]/rows/upsert",
-    "operation": "tables.rows.upsert"
+    "path": "/api/v2/tools/[toolId]/execute",
+    "operation": "tools.execute"
   },
   {
     "method": "POST",

apps/sim/lib/api/server/routes/copilot-refused-routes.json

@@ -14,11 +14,6 @@
     "path": "/api/v2/files/[fileId]/versions/[version]/content",
     "operation": "files.versions.download"
   },
-  {
-    "method": "GET",
-    "path": "/api/v2/knowledge/[knowledgeBaseId]/export",
-    "operation": "knowledge.export"
-  },
   {
     "method": "GET",
     "path": "/api/v2/meta",