fix(mothership): close sharing, line-window and snapshot gaps in content delivery

Chat may share or see links only to clean files whatever the surface, text
is redacted before a line window cuts it, accounting follows a decision,
cached compiles and knowledge metadata bind to the versions they show,
row-search matches classify at their matched revision, and table counts
follow admitted rows.

Author: waleedlatif1

.claude/rules/sim-content-delivery.md

@@ -38,8 +38,10 @@ the workflow executor see byte-identical results.
     redaction reaches: binary, base64, archives, images, public links).
 - Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries
   into the registry egress projects with; the **session gate** (run_code sandbox proxy) records
-  them into the workbench history. Neither ever admits `unknown`, and one refused item never
-  latches the shared registry.
+  them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct
+  single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's
+  registry so the whole output is withheld and the workbench copy stays unknown. It needs a
+  registry to latch; without one it refuses. One refused item never latches the shared registry.
 - `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches
   the model (the user's own edit preview; a derived file that carries its sources' provenance).
   Name why at each use.
@@ -73,7 +75,7 @@ own unless noted.
 | Bulk download zip | Left out | Left out (opaque) | Included |
 | `files search` | Results dropped | Results dropped (a match is an oracle) | Returned |
 | File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
-| Chat public share activation | 403 | 403 (a link is opaque) | Shares |
+| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
 | Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
 | Doc compile | Compile refused | Error text redacted | Compiles |
 | Run file download | 403 | Redacted | Delivered |
@@ -82,5 +84,5 @@ own unless noted.
 | KB export, table export download | Not available to Chat | | |
 | Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
 | Row search | Match dropped | Match in a secret column dropped | Returned |
-| Table import from a file | Rows stored unknown | Per-cell entries | Exact-empty, as before |
+| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
 | `tools execute` for file, knowledge, table, memory, function, custom, MCP tools | Not available to Chat | | |

.cursor/rules/sim-content-delivery.mdc

@@ -29,8 +29,10 @@ the workflow executor see byte-identical results.
     redaction reaches: binary, base64, archives, images, public links).
 - Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries
   into the registry egress projects with; the **session gate** (run_code sandbox proxy) records
-  them into the workbench history. Neither ever admits `unknown`, and one refused item never
-  latches the shared registry.
+  them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct
+  single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's
+  registry so the whole output is withheld and the workbench copy stays unknown. It needs a
+  registry to latch; without one it refuses. One refused item never latches the shared registry.
 - `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches
   the model (the user's own edit preview; a derived file that carries its sources' provenance).
   Name why at each use.
@@ -64,7 +66,7 @@ own unless noted.
 | Bulk download zip | Left out | Left out (opaque) | Included |
 | `files search` | Results dropped | Results dropped (a match is an oracle) | Returned |
 | File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged |
-| Chat public share activation | 403 | 403 (a link is opaque) | Shares |
+| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares |
 | Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered |
 | Doc compile | Compile refused | Error text redacted | Compiles |
 | Run file download | 403 | Redacted | Delivered |
@@ -73,5 +75,5 @@ own unless noted.
 | KB export, table export download | Not available to Chat | | |
 | Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered |
 | Row search | Match dropped | Match in a secret column dropped | Returned |
-| Table import from a file | Rows stored unknown | Per-cell entries | Exact-empty, as before |
+| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before |
 | `tools execute` for file, knowledge, table, memory, function, custom, MCP tools | Not available to Chat | | |

apps/sim/lib/core/application/content-delivery.test.ts

@@ -246,6 +246,21 @@ describe('runDeclaredContentOperation', () => {
     expect(record.gateFailure).toBe(failure)
   })
 
+  it('does not account for a result whose admission failed and was caught', async () => {
+    const failing: ContentDeliveryGate = {
+      async admit() {
+        throw new Error('gate down')
+      },
+    }
+    const outcome = await observeContentDelivery(failing, () =>
+      runDeclaredContentOperation(SOURCED, CHAT, async () => {
+        await admitContent(CHAT, [source('read')]).catch(() => undefined)
+        return 'content returned anyway'
+      }).catch((error: unknown) => error)
+    )
+    expect(outcome).toBeInstanceOf(ContentDeliveryUnaccountedError)
+  })
+
   it('does not let a nested use case account for what its enclosing operation returns', async () => {
     const admitAll: ContentDeliveryGate = {
       async admit(sources) {

apps/sim/lib/core/application/content-delivery.ts

@@ -40,6 +40,11 @@ export interface ContentSource {
  */
 export interface ContentDeliveryGate {
   admit(sources: readonly ContentSource[]): Promise<readonly boolean[]>
+  /**
+   * The text as this surface's egress will show it, or `undefined` when it cannot be shown. A gate
+   * whose surface redacts (Chat's registry) implements it; see {@link redactBeforeCutting}.
+   */
+  project?(text: string): string | undefined
 }
 
 /**
@@ -135,6 +140,24 @@ export function observeContentDelivery<T>(gate: ContentDeliveryGate, execute: ()
   return gates.run(gate, execute)
 }
 
+/**
+ * Text a use case is about to cut (a line window, a clip), redacted first by the surface's gate.
+ * Egress redacts whole secrets only, so a cut through one would leave a fragment it cannot
+ * recognize. Other callers, and surfaces that do not redact, get the text unchanged.
+ */
+export function redactBeforeCutting(
+  principal: Principal | undefined,
+  text: string,
+  kind: ContentSourceKind
+): string {
+  if (principal && !isCopilotPrincipal(principal)) return text
+  const project = gates.getStore()?.project
+  if (!project) return text
+  const projected = project(text)
+  if (projected === undefined) throw new ContentRefusedError(kind)
+  return projected
+}
+
 /** Admits every item: see {@link observeContentOutsideModel}. */
 const OUTSIDE_MODEL_GATE: ContentDeliveryGate = {
   async admit(sources) {
@@ -222,8 +245,9 @@ export async function admitContent(
   ) {
     throw new ContentDeliveryUndeclaredError()
   }
-  markAccounted()
+  /** Accounting follows a decision: a caller that catches a failure has classified nothing. */
   if (sources.length === 0 || !mustClassifyContent(principal)) {
+    markAccounted()
     return new ContentAdmission(
       sources,
       sources.map(() => true)
@@ -242,6 +266,7 @@ export async function admitContent(
     }
     throw error
   }
+  markAccounted()
   if (ledger) {
     sources.forEach((source, index) => {
       if (verdicts[index]) {
@@ -274,6 +299,8 @@ export async function admitWrittenContent(
      */
     const [first] = sources
     if (!ledgers.getStore() || !first) throw error
+    /** The result is decided after all: withheld whole, as saved-not-shown. */
+    markAccounted()
     withholdAfterWrite(first.kind)
     return new ContentAdmission(
       sources,

apps/sim/lib/knowledge/application/admit-knowledge-content.ts

@@ -3,6 +3,8 @@ import { admitEach, type ContentSourceKind } from '@/lib/core/application/conten
 import {
   classifyKnowledgeChunks,
   classifyKnowledgeDocumentMetadata,
+  type KnowledgeDocumentSnapshot,
+  knowledgeDocumentSnapshot,
 } from '@/lib/knowledge/secret-provenance'
 
 interface KnowledgeAdmissionOptions {
@@ -16,18 +18,28 @@ const writtenAs = (options: KnowledgeAdmissionOptions, kind: ContentSourceKind)
 /**
  * Admits each document whose rendered metadata a result returns, on its own, and returns the ids
  * admitted. `value` is what the result shows of the document, so only secrets it contains are
- * imported. A caller no gate observes gets every id back without a provenance read.
+ * imported; `snapshot` names the stored fields it shows, when they are not its own keys. A caller
+ * no gate observes gets every id back without a provenance read.
  */
 export async function admitKnowledgeDocuments(
   principal: Principal,
-  documents: readonly { id: string; value: unknown }[],
+  documents: readonly { id: string; value: unknown; snapshot?: KnowledgeDocumentSnapshot }[],
   options: KnowledgeAdmissionOptions = {}
 ): Promise<Set<string>> {
   const admitted = await admitEach(
     principal,
     documents,
     async (items) => {
-      const provenance = await classifyKnowledgeDocumentMetadata(items.map((item) => item.id))
+      /** Each classification is bound to the metadata the item shows; see the classifier. */
+      const snapshots = new Map<string, KnowledgeDocumentSnapshot[]>()
+      for (const item of items) {
+        const snapshot = item.snapshot ?? knowledgeDocumentSnapshot(item.value)
+        if (snapshot) snapshots.set(item.id, [...(snapshots.get(item.id) ?? []), snapshot])
+      }
+      const provenance = await classifyKnowledgeDocumentMetadata(
+        items.map((item) => item.id),
+        snapshots
+      )
       return items.map((item) => ({
         kind: 'knowledge-document' as const,
         id: item.id,

apps/sim/lib/knowledge/application/chunks.test.ts

@@ -49,7 +49,11 @@ import { admitContent } from '@/lib/core/application/content-delivery'
 import { ForbiddenOperationError } from '@/lib/core/application/forbidden'
 import { WORKSPACE_ACCESS_SCOPE } from '@/lib/knowledge/access/scope'
 import { KnowledgeDocumentNotReadyError } from '@/lib/knowledge/application/chunk-errors'
-import { bulkUpdateKnowledgeChunks, listKnowledgeChunks } from '@/lib/knowledge/application/chunks'
+import {
+  bulkUpdateKnowledgeChunks,
+  createKnowledgeChunk,
+  listKnowledgeChunks,
+} from '@/lib/knowledge/application/chunks'
 
 describe('knowledge chunk application use cases', () => {
   beforeEach(() => {
@@ -75,6 +79,37 @@ describe('knowledge chunk application use cases', () => {
     })
   }
 
+  it('admits a created chunk with the document name its response shows', async () => {
+    workspaceAuthzMockFns.mockResolveEffectiveWorkspacePermission.mockResolvedValue('write')
+    knowledgeContextsMockFns.mockResolveCanonicalActiveKnowledgeDocumentContext.mockResolvedValue({
+      workspaceId: 'workspace-1',
+      workspaceOrganizationId: null,
+      allowPersonalApiKeys: true,
+      billedAccountUserId: 'billing-owner-1',
+      access: { get: async () => WORKSPACE_ACCESS_SCOPE },
+      knowledgeBaseId: 'knowledge-1',
+      knowledgeBase: { id: 'knowledge-1', embeddingModel: 'text-embedding-3-small' },
+      documentId: 'document-1',
+      document: { id: 'document-1', processingStatus: 'completed', filename: 'named.txt' },
+    })
+    mocks.createChunk.mockResolvedValue({ id: 'chunk-new', content: 'body', tokenCount: 1 })
+    admitOnly(['chunk-new'])
+
+    const result = await createKnowledgeChunk.execute({
+      principal: createSessionPrincipal(),
+      input: {
+        knowledgeBaseId: 'knowledge-1',
+        documentId: 'document-1',
+        content: 'body',
+        resolveContentProvenance: () => undefined,
+      },
+    })
+
+    expect(result.chunk.documentName).toBe('named.txt')
+    const [, [item]] = mocks.admitKnowledgeChunks.mock.calls[0]
+    expect(item.value).toMatchObject({ documentId: 'document-1', documentName: 'named.txt' })
+  })
+
   describe('a searched chunk listing', () => {
     const page = {
       chunks: [

apps/sim/lib/knowledge/application/chunks.ts

@@ -75,12 +75,14 @@ async function requireAdmittedChunk(
   principal: Principal,
   documentId: string,
   chunk: { id: string; content: string },
-  options: { written?: boolean } = {}
+  options: { written?: boolean; shown?: Record<string, unknown> } = {}
 ): Promise<void> {
+  /** `shown` is every other field the response adds beside the chunk, such as its document name. */
+  const value = options.shown ? { ...chunk, ...options.shown } : chunk
   const admitted = await admitKnowledgeChunks(
     principal,
-    [{ id: chunk.id, documentId, content: chunk.content, value: chunk }],
-    options
+    [{ id: chunk.id, documentId, content: chunk.content, value }],
+    { written: options.written }
   )
   if (!options.written && !admitted.has(chunk.id)) throw new ContentRefusedError('knowledge-chunk')
 }
@@ -239,7 +241,10 @@ export const createKnowledgeChunk = defineAuthorizedKnowledgeUseCase({
     } catch (error) {
       logger.warn('Failed to calculate cost for chunk upload', { error })
     }
-    await requireAdmittedChunk(principal, context.documentId, chunk, { written: true })
+    await requireAdmittedChunk(principal, context.documentId, chunk, {
+      written: true,
+      shown: { documentId: context.documentId, documentName: context.document.filename },
+    })
     return {
       chunk: {
         ...chunk,

apps/sim/lib/knowledge/application/tags.test.ts

@@ -396,6 +396,31 @@ describe('knowledge tag application use cases', () => {
       )
     })
 
+    it('answers Chat an empty usage for a base with no tag definitions', async () => {
+      mocks.readDetailedUsage.mockResolvedValue([])
+      await expect(
+        readDetailedKnowledgeTagUsage.execute({
+          principal: chat,
+          input: { knowledgeBaseId: 'knowledge-b' },
+        })
+      ).resolves.toMatchObject({ usage: [] })
+      expect(admission.admitKnowledgeDocuments).not.toHaveBeenCalled()
+    })
+
+    it('binds each occurrence to the stored tag value it shows', async () => {
+      await readDetailedKnowledgeTagUsage.execute({
+        principal: chat,
+        input: { knowledgeBaseId: 'knowledge-b' },
+      })
+      const [, documents] = admission.admitKnowledgeDocuments.mock.calls[1]
+      expect(documents).toEqual([
+        expect.objectContaining({
+          id: 'doc-shared',
+          snapshot: { filename: 'shared.csv', tag2: 'ops' },
+        }),
+      ])
+    })
+
     it('decides each (definition, document) occurrence on its own', async () => {
       const result = await readDetailedKnowledgeTagUsage.execute({
         principal: chat,

apps/sim/lib/knowledge/application/tags.ts

@@ -363,11 +363,17 @@ async function admitTagUsage(
   principal: Principal,
   usage: Awaited<ReturnType<typeof getTagUsage>>
 ): Promise<Awaited<ReturnType<typeof getTagUsage>>> {
+  /** A base with no tag definitions returns nothing, which is still an accounting. */
+  if (usage.length === 0) await admitContent(principal, [])
   const admitted = []
   for (const definition of usage) {
     const ids = await admitKnowledgeDocuments(
       principal,
-      definition.documents.map((doc) => ({ id: doc.id, value: doc }))
+      definition.documents.map((doc) => ({
+        id: doc.id,
+        value: doc,
+        snapshot: { filename: doc.name, [definition.tagSlot]: doc.tagValue },
+      }))
     )
     const documents = definition.documents.filter((doc) => ids.has(doc.id))
     admitted.push(

apps/sim/lib/knowledge/secret-provenance.test.ts

@@ -5,9 +5,11 @@ import { beforeEach, describe, expect, it, vi } from 'vitest'
 import type { DbTransaction } from '@/lib/db/types'
 import type { DurableSecretProvenance } from '@/lib/execution/durable-secret-provenance'
 import {
+  classifyKnowledgeDocumentMetadata,
   createKnowledgeDocumentSourceValue,
   importKnowledgePersistedResponseSecretProvenance,
   importKnowledgeSearchResultSecretProvenance,
+  knowledgeDocumentSnapshot,
   loadKnowledgeDocumentSecretRegistry,
   replaceKnowledgeDocumentSecretProvenanceInTx,
 } from '@/lib/knowledge/secret-provenance'
@@ -167,3 +169,50 @@ describe('knowledge durable provenance enforcement', () => {
     expect(registry.isPermanentlyIncomplete()).toBe(true)
   })
 })
+
+describe('document metadata classification bound to the shown snapshot', () => {
+  const NUMBERED_ROW = { ...DOCUMENT_ROW, tag1: 'emea', number1: 42 }
+
+  beforeEach(() => {
+    resetDbChainMock()
+  })
+
+  it('keeps the stored classification while the shown metadata is what is stored', async () => {
+    queueTableRows(document, [NUMBERED_ROW])
+    const result = await classifyKnowledgeDocumentMetadata(
+      [NUMBERED_ROW.id],
+      new Map([[NUMBERED_ROW.id, [{ filename: 'source.pdf', tag1: 'emea', number1: '42' }]]])
+    )
+    expect(result.get(NUMBERED_ROW.id)?.status).toBe('exact')
+  })
+
+  it.each([
+    ['a renamed document', { filename: 'previous-name.pdf' }],
+    ['an edited tag', { tag1: 'previous value' }],
+    ['a tag cleared since', { tag2: 'previous value' }],
+    ['an edited number', { number1: '7' }],
+  ])(
+    'reads %s as unknown, so stale values are not paired with new provenance',
+    async (_, shown) => {
+      queueTableRows(document, [NUMBERED_ROW])
+      const result = await classifyKnowledgeDocumentMetadata(
+        [NUMBERED_ROW.id],
+        new Map([[NUMBERED_ROW.id, [shown]]])
+      )
+      expect(result.get(NUMBERED_ROW.id)).toEqual({ status: 'unknown' })
+    }
+  )
+
+  it('picks only the editable metadata a returned value shows', () => {
+    expect(
+      knowledgeDocumentSnapshot({
+        id: 'd',
+        filename: 'f.pdf',
+        tag3: 'x',
+        fileUrl: '/u',
+        date1: 'd',
+      })
+    ).toEqual({ filename: 'f.pdf', tag3: 'x' })
+    expect(knowledgeDocumentSnapshot({ id: 'd', fileUrl: '/u' })).toBeUndefined()
+  })
+})