diff --git a/.claude/rules/sim-content-delivery.md b/.claude/rules/sim-content-delivery.md new file mode 100644 index 00000000000..6d976d8471d --- /dev/null +++ b/.claude/rules/sim-content-delivery.md @@ -0,0 +1,104 @@ +--- +description: How stored content (workspace files, run files, knowledge, table rows) reaches Chat's model, and the per-path policy. Read before changing a use case, route or Chat surface that returns such content. +paths: + - "apps/sim/lib/core/application/**" + - "apps/sim/lib/mothership/content-delivery/**" + - "apps/sim/lib/workspace-files/application/**" + - "apps/sim/lib/knowledge/application/**" + - "apps/sim/lib/table/application/**" + - "apps/sim/lib/workflows/application/**" + - "apps/sim/lib/mothership/chat/application/**" + - "apps/sim/lib/**/application/operations.ts" + - "apps/sim/lib/mothership/agent-cli/**" + - "apps/sim/lib/mothership/tools/**" + - "apps/sim/app/api/v2/**" +--- + +# Sim Content Delivery to Chat + +Content derived from a provenance-bearing store reaches Chat's model only after **each item** it +came from is admitted by the gate of the surface it leaves through. The stores are workspace files +(and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers +bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output, +error text that quotes content, and public links. Counts, ordering, and match locations are ordinary +metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional +human uploads remain ordinary input and are not scanned. + +Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are +classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and +the workflow executor see byte-identical results. + +## The pieces + +- `lib/core/application/content-delivery.ts`: the primitive. + - `admitContent(principal, sources)` returns per-item verdicts. `requireAdmittedContent` is the + single-item form and throws `ContentRefusedError` (403). + - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure + does not undo the write: audit and effects run, then the operation answers with the refusal. + Settled tool results use the same deferral, but do not claim that a mutation was saved. + - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry + may appear), `{ derived: value }` (only entries occurring in `value`), `'opaque'` (bytes no + redaction reaches: binary, base64, archives, images, public links), or `'excerpt'` (text already + cut before admission, where a secret fragment cannot be redacted). Both gates withhold an + excerpt with known secret entries; raw opaque bytes may still enter the workbench with history. + - `redactBeforeCutting` projects complete text before selecting a line window. The session gate + creates the existing redactor from the operation's bounded evidence only when a cut needs it; + ordinary byte reads do not decrypt a second time or reload history. +- Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries + into the registry egress projects with; the **session gate** (run_code sandbox proxy) records + them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct + single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's + registry so the whole output is withheld and the workbench copy stays unknown. It needs a + registry to latch; without one it refuses. One refused item never latches the shared registry. +- `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches + the model (the user's own edit preview; a derived file that carries its sources' provenance). + Name why at each use. + +## Rules + +1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result + can carry content from a store, `'none'` otherwise. Definition throws + without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one. +2. **A `sourced` use case accounts for what it returns**, before it loads content: call + `admitContent` (with `[]` when this call returns none). Otherwise Chat gets + `ContentDeliveryUnaccountedError`, and the transports turn the response into a 403. +3. **Classify before loading bytes**, and exclude refused items per item. Withhold a whole result + only when its items cannot be separated (one file, one render, one compile). +4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.** + `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive. + Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`. +5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for + Chat goes through the gated use case, or names `observeContentOutsideModel` and why. + +## Policy + +U = `unknown`, E = exact with secret entries, R/C = unrecorded or clean. Each item is decided on its +own unless noted. + +| Path | U | E | R / C | +|---|---|---|---| +| File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered | +| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered | +| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered | +| File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered | +| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included | +| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned | +| File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged | +| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares | +| Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered | +| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references | +| Run file download | 403 | Redacted | Delivered | +| Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) | +| KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered | +| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source | +| Table export download | Existing unsupported Chat operation | | | +| Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered | +| Row search | Locations returned without cell values | Same | Returned | +| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before | +| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization | + +Search pagination and aggregate totals retain their database meaning even when an item or preview is +withheld. Do not scan unrelated table rows or whole document collections to conceal counts. +A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path. +Registry admission commits only when the complete item fits the existing capacity; a refused item +must not consume capacity or permanently disable the parent registry. diff --git a/.cursor/rules/sim-content-delivery.mdc b/.cursor/rules/sim-content-delivery.mdc new file mode 100644 index 00000000000..4e1b90df027 --- /dev/null +++ b/.cursor/rules/sim-content-delivery.mdc @@ -0,0 +1,95 @@ +--- +description: "How stored content (workspace files, run files, knowledge, table rows) reaches Chat's model, and the per-path policy. Read before changing a use case, route or Chat surface that returns such content." +globs: ["apps/sim/lib/core/application/**","apps/sim/lib/mothership/content-delivery/**","apps/sim/lib/workspace-files/application/**","apps/sim/lib/knowledge/application/**","apps/sim/lib/table/application/**","apps/sim/lib/workflows/application/**","apps/sim/lib/mothership/chat/application/**","apps/sim/lib/**/application/operations.ts","apps/sim/lib/mothership/agent-cli/**","apps/sim/lib/mothership/tools/**","apps/sim/app/api/v2/**"] +--- + + + +# Sim Content Delivery to Chat + +Content derived from a provenance-bearing store reaches Chat's model only after **each item** it +came from is admitted by the gate of the surface it leaves through. The stores are workspace files +(and their versions), run files, knowledge documents and chunks, and table rows. "Derived" covers +bytes, text, excerpts, chunks, cells, previews, base64, archives, rendered or compiled output, +error text that quotes content, and public links. Counts, ordering, and match locations are ordinary +metadata; this boundary protects supported literal secret values, not arbitrary inference. Intentional +human uploads remain ordinary input and are not scanned. + +Only Copilot principals (`delegated` / `organization_delegated` with `serviceId: 'copilot'`) are +classified. Every other caller is admitted without a provenance read, so the UI, API keys, MCP and +the workflow executor see byte-identical results. + +## The pieces + +- `lib/core/application/content-delivery.ts`: the primitive. + - `admitContent(principal, sources)` returns per-item verdicts. `requireAdmittedContent` is the + single-item form and throws `ContentRefusedError` (403). + - `admitWrittenContent` is for a committed write that returns content. A refusal or gate failure + does not undo the write: audit and effects run, then the operation answers with the refusal. + Settled tool results use the same deferral, but do not claim that a mutation was saved. + - A source is `{ kind, id, provenance, view }`. `view` is `'complete'` (text whose every entry + may appear), `{ derived: value }` (only entries occurring in `value`), `'opaque'` (bytes no + redaction reaches: binary, base64, archives, images, public links), or `'excerpt'` (text already + cut before admission, where a secret fragment cannot be redacted). Both gates withhold an + excerpt with known secret entries; raw opaque bytes may still enter the workbench with history. + - `redactBeforeCutting` projects complete text before selecting a line window. The session gate + creates the existing redactor from the operation's bounded evidence only when a cut needs it; + ordinary byte reads do not decrypt a second time or reload history. +- Gates, installed by the surface: the **registry gate** (Chat tools, CLI, mounts) imports entries + into the registry egress projects with; the **session gate** (run_code sandbox proxy) records + them into the workbench history. Neither admits `unknown`, with one exception: the CLI's direct + single-file download (`files get`) admits it under `unknown: 'taint'`, which latches the call's + registry so the whole output is withheld and the workbench copy stays unknown. It needs a + registry to latch; without one it refuses. One refused item never latches the shared registry. +- `observeContentOutsideModel` reads for Chat without classifying, for content that never reaches + the model (the user's own edit preview; a derived file that carries its sources' provenance). + Name why at each use. + +## Rules + +1. **Every operation that admits Copilot declares `contentDelivery`**: `'sourced'` when its result + can carry content from a store, `'none'` otherwise. Definition throws + without it, and `copilot-route-inventory.test.ts` lists every Chat-reachable route without one. +2. **A `sourced` use case accounts for what it returns**, before it loads content: call + `admitContent` (with `[]` when this call returns none). Otherwise Chat gets + `ContentDeliveryUnaccountedError`, and the transports turn the response into a 403. +3. **Classify before loading bytes**, and exclude refused items per item. Withhold a whole result + only when its items cannot be separated (one file, one render, one compile). +4. **A new `sourced` route needs a case in `lib/mothership/__integration__/content-delivery.integration.ts`.** + `copilot-content-routes.json` lists them; the suite fails for a listed route it does not drive. + Refresh the list with `vitest run lib/api/server/routes/copilot-route-inventory.test.ts --update=all`. +5. **Do not reach a store's bytes around a use case for Chat.** A helper that reads content for + Chat goes through the gated use case, or names `observeContentOutsideModel` and why. + +## Policy + +U = `unknown`, E = exact with secret entries, R/C = unrecorded or clean. Each item is decided on its +own unless noted. + +| Path | U | E | R / C | +|---|---|---|---| +| File download (CLI `files get`) | Delivered once; invocation withheld from the model; workbench copy stays unknown | Redacted | Delivered | +| File download (run_code proxy) | Refused before bytes enter workbench history | Recorded in workbench history | Delivered | +| Requested mounts | Refused before code executes, listed in `excludedInputs`; later calls remain usable | Recorded in workbench history | Delivered | +| File text, version text, `files read`, grep | 403 | Redacted (grep redacts before clipping) | Delivered | +| Bulk download zip | Left out, reported in the archive manifest | Left out (opaque), reported in the archive manifest | Included | +| `files search` | Match location kept; clipped preview withheld | Same; read/grep provides safely redacted text | Returned | +| File edit (`PATCH` content) | 403 before reading | Diagnostics redacted | Unchanged | +| Chat public share activation, and seeing an existing link (any surface, run_code included) | 403 | 403 (a link leaves every gate) | Shares | +| Rendered view, streamed image, inline attachments | Refused | Refused (opaque) | Delivered | +| Doc compile | Actual file read refused | Evidence imported before broker bytes/errors return | Compiles, including dynamic file references | +| Run file download | 403 | Redacted | Delivered | +| Run read / execute / resume `base64` | `base64` removed, descriptor kept | Same | Inlined (sync runs only) | +| KB chunks, search, documents, tags | Item dropped (single read: 403) | Redacted | Delivered | +| KB export | Stream fails before refused source bytes; no successful archive receipt | Refused (opaque) | Streams originals/chunks with admission at each source | +| Table export download | Existing unsupported Chat operation | | | +| Table rows, query, read, `table_selection` | Row dropped (single read: 403) | Redacted | Delivered | +| Row search | Locations returned without cell values | Same | Returned | +| Table import from a file | Rows stored unknown | Per-cell entries; a row whose coercion rewrote a secret, or whose raw cells cannot be checked, is stored unknown | Exact-empty, as before | +| `tools execute` | Settled result withheld, with no automatic retry | Named masking through the existing registry | Returned, subject to existing tool authorization | + +Search pagination and aggregate totals retain their database meaning even when an item or preview is +withheld. Do not scan unrelated table rows or whole document collections to conceal counts. +A refused requested mount stops code before it runs, avoiding an older file at the same sandbox path. +Registry admission commits only when the complete item fits the existing capacity; a refused item +must not consume capacity or permanently disable the parent registry. diff --git a/apps/sim/AGENTS.md b/apps/sim/AGENTS.md index 9da2a087205..8f3822fa977 100644 --- a/apps/sim/AGENTS.md +++ b/apps/sim/AGENTS.md @@ -9,6 +9,7 @@ Applies to `apps/sim/**` on top of the root [AGENTS.md](/AGENTS.md), which holds - Styling and emcn chrome, UI copy, settings pages: `sim-styling.md`, `sim-ui-copy.md`, `sim-settings-pages.md` - URL state, list and menu order, caching: `sim-url-state.md`, `sim-list-ordering.md`, `sim-caching.md` - Tools, blocks, triggers: `sim-integrations.md`; isolated-vm sandbox worker: `sim-sandbox.md` +- Anything that returns stored file, run-file, knowledge or table content to Chat: `sim-content-delivery.md` - Tests: `sim-testing.md` and the `test-audit` skill - Landing pages: `app/(landing)/CLAUDE.md`, `landing-seo-geo.md`, `constitution.md` (product language) diff --git a/apps/sim/app/api/v2/files/bulk-download/route.test.ts b/apps/sim/app/api/v2/files/bulk-download/route.test.ts index 2e98a0b3ce5..b2352f3c29b 100644 --- a/apps/sim/app/api/v2/files/bulk-download/route.test.ts +++ b/apps/sim/app/api/v2/files/bulk-download/route.test.ts @@ -76,6 +76,7 @@ describe('GET /api/v2/files/bulk-download', () => { folderPaths: new Map(), renderedDocuments: new Map(), declaredBytes: 6, + excludedFileIds: [], }) }) diff --git a/apps/sim/app/api/v2/files/bulk-download/route.ts b/apps/sim/app/api/v2/files/bulk-download/route.ts index f2322d778ce..5587caa1a16 100644 --- a/apps/sim/app/api/v2/files/bulk-download/route.ts +++ b/apps/sim/app/api/v2/files/bulk-download/route.ts @@ -54,14 +54,16 @@ export const GET = defineV2BinaryRoute({ folderPaths: query.folderPaths, }), useCase: downloadWorkspaceFileItems, - present: ({ filesToZip, folderPaths, renderedDocuments }) => { - const entryPaths = buildZipEntryPaths( - filesToZip.map((file) => ({ + present: ({ filesToZip, folderPaths, renderedDocuments, excludedFileIds }) => { + const report = excludedFileIds.length > 0 ? { partial: true, excludedFileIds } : undefined + const entryPaths = buildZipEntryPaths([ + ...filesToZip.map((file) => ({ name: file.name, folderPath: file.folderId ? folderPaths.get(file.folderId) : null, contentType: file.type, - })) - ) + })), + ...(report ? [{ name: 'sim-download-report.json', contentType: 'application/json' }] : []), + ]) const archive = new ZipArchive({ store: true }) archive.on('warning', (error: Error) => { logger.warn('Archive warning while streaming workspace files', { error }) @@ -71,6 +73,9 @@ export const GET = defineV2BinaryRoute({ name: entryPaths[index], }) }) + if (report) { + archive.append(JSON.stringify(report, null, 2), { name: entryPaths[filesToZip.length] }) + } archive.finalize().catch((error) => { logger.error('Failed to finalize workspace file archive', { error }) }) diff --git a/apps/sim/app/api/v2/lib/response.ts b/apps/sim/app/api/v2/lib/response.ts index 30eaa7bb044..6b0db3d312e 100644 --- a/apps/sim/app/api/v2/lib/response.ts +++ b/apps/sim/app/api/v2/lib/response.ts @@ -10,6 +10,7 @@ import { type CursorKey, INVALID_CURSOR_MESSAGE } from '@/lib/api/list-query' import { getValidationErrorMessage, serializeZodIssues } from '@/lib/api/server' import { ADMISSION_RETRY_AFTER_SECONDS } from '@/lib/core/admission/transient-failure' import { + ContentDeliveryUnobservedError, forbiddenErrorDetails, InsufficientScopeError, OAuthAccessTokenExpiredError, @@ -530,6 +531,10 @@ export function v2ErrorForOrchestration( * one its failure class maps to. */ export function v2CaughtOrchestrationError(error: unknown): NextResponse | null { + /** Chat reached stored content outside any classifying surface: transient, never a content leak. */ + if (error instanceof ContentDeliveryUnobservedError) { + return v2Error('SERVICE_UNAVAILABLE', error.message) + } const classified = asOrchestrationError(error) if (!classified) return null if (classified instanceof InsufficientScopeError) return v2InsufficientScope(classified) diff --git a/apps/sim/ee/access-requests/lib/application/operations.ts b/apps/sim/ee/access-requests/lib/application/operations.ts index 0af98a38bd9..13a313be0cc 100644 --- a/apps/sim/ee/access-requests/lib/application/operations.ts +++ b/apps/sim/ee/access-requests/lib/application/operations.ts @@ -51,6 +51,7 @@ function defineAccessRequestOperation( * permission-group-exempt: requests never grant a withheld capability without administrator review. */ capability: 'none', + contentDelivery: 'none', }) return Object.freeze({ ...workspaceOperation, organizationOperation, admin }) } diff --git a/apps/sim/ee/workspace-forking/application/operations.ts b/apps/sim/ee/workspace-forking/application/operations.ts index e744d99f92f..878930fe49d 100644 --- a/apps/sim/ee/workspace-forking/application/operations.ts +++ b/apps/sim/ee/workspace-forking/application/operations.ts @@ -14,6 +14,7 @@ export const forkOperations = { discover: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.discover', oauthScope: 'api:read', }), @@ -23,6 +24,7 @@ export const forkOperations = { preview: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.preview', oauthScope: 'api:write', }), @@ -32,6 +34,7 @@ export const forkOperations = { create: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.create', oauthScope: 'api:write', }), @@ -41,6 +44,7 @@ export const forkOperations = { syncPreview: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.sync.preview', oauthScope: 'api:write', }), @@ -50,6 +54,7 @@ export const forkOperations = { sync: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.sync', oauthScope: 'api:write', }), @@ -59,6 +64,7 @@ export const forkOperations = { mappingsRead: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.mappings.read', oauthScope: 'api:read', }), @@ -68,6 +74,7 @@ export const forkOperations = { mappingsUpdate: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.mappings.update', oauthScope: 'api:write', }), @@ -77,6 +84,7 @@ export const forkOperations = { rollback: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.rollback', oauthScope: 'api:write', }), @@ -86,6 +94,7 @@ export const forkOperations = { unlink: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.unlink', oauthScope: 'api:write', }), @@ -95,6 +104,7 @@ export const forkOperations = { exclusions: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.exclusions', oauthScope: 'api:write', }), @@ -107,6 +117,7 @@ export const forkOperations = { syncDefault: defineWorkspaceOperation({ ...adminPolicy, capability: 'none', + contentDelivery: 'none', id: 'workspaces.fork.sync_default', oauthScope: 'api:write', }), diff --git a/apps/sim/executor/utils/resolved-secret-content-projection.ts b/apps/sim/executor/utils/resolved-secret-content-projection.ts index bc7eb1f256f..be5d156c03c 100644 --- a/apps/sim/executor/utils/resolved-secret-content-projection.ts +++ b/apps/sim/executor/utils/resolved-secret-content-projection.ts @@ -93,6 +93,8 @@ interface ProjectionState { } export interface ResolvedSecretContentProjectionOptions { + /** Keeps original logical lines when masking complete text before a file line window. */ + preserveLineBreaks?: boolean /** Values already materialized and verified by a boundary-specific projector. */ isOpaqueSafeObject?: (value: object) => boolean /** Whether string-shaped secret literals should also match typed number/boolean/null values. */ @@ -304,13 +306,13 @@ function sanitizeProjectedString( ? sanitizeCollisionProneInternalDiagnosticIdentifiers(value, matcher) : value sanitized = matcher - ? sanitizeResolvedSecretString(sanitized, matcher, maxBytes, options.onMatch) + ? sanitizeResolvedSecretString(sanitized, matcher, maxBytes, options.onMatch, options) : sanitized if (options.sanitizeInternalIdentifiers) { sanitized = sanitizeInternalDiagnosticIdentifiers(sanitized) } if (matcher && containsResolvedSecret(sanitized, matcher)) { - sanitized = sanitizeResolvedSecretString(sanitized, matcher, maxBytes, options.onMatch) + sanitized = sanitizeResolvedSecretString(sanitized, matcher, maxBytes, options.onMatch, options) if (containsResolvedSecret(sanitized, matcher)) { throw new ResolvedSecretContentProjectionError( 'Secret replacement could not produce safe model content' diff --git a/apps/sim/executor/utils/resolved-secret-matcher.test.ts b/apps/sim/executor/utils/resolved-secret-matcher.test.ts index 46f37faac0d..cf3fa96188e 100644 --- a/apps/sim/executor/utils/resolved-secret-matcher.test.ts +++ b/apps/sim/executor/utils/resolved-secret-matcher.test.ts @@ -11,6 +11,31 @@ import { const PRESERVE_NAMED_PROVENANCE = { preserveNamedProvenanceLabels: true } as const describe('resolved secret matcher', () => { + it.each([ + ['first-line\nsecond-line', '{{TOKEN}}\n{{TOKEN}}'], + ['first-line\r\nsecond-line', '{{TOKEN}}\r\n{{TOKEN}}'], + ['first-line\nsecond-line\n', '{{TOKEN}}\n{{TOKEN}}\n'], + ['first-line\n\nsecond-line', '{{TOKEN}}\n{{TOKEN}}\n{{TOKEN}}'], + ['\n'.repeat(8), '{{TOKEN}}\n'.repeat(8)], + ])( + 'preserves line boundaries for a complete literal %j when selecting file lines', + (plaintext, expected) => { + const matcher = createResolvedSecretMatcher([{ plaintext, replacement: '{{TOKEN}}' }]) + if (!matcher) throw new Error('Fixture literal did not produce a matcher') + expect( + sanitizeResolvedSecretString(plaintext, matcher, undefined, undefined, { + preserveLineBreaks: true, + }) + ).toBe(expected) + expect(sanitizeResolvedSecretString(plaintext, matcher)).toBe('{{TOKEN}}') + expect( + sanitizeResolvedSecretString(`before ${plaintext} after`, matcher, undefined, undefined, { + preserveLineBreaks: true, + }) + ).toBe(`before ${expected} after`) + } + ) + it('reports each matched literal once across large repeated content', () => { const matcher = createResolvedSecretMatcher([ { plaintext: 'xxxxxxxx', replacement: '{{SHORT}}' }, diff --git a/apps/sim/executor/utils/resolved-secret-matcher.ts b/apps/sim/executor/utils/resolved-secret-matcher.ts index 7ea20237747..2cafdb9cdb2 100644 --- a/apps/sim/executor/utils/resolved-secret-matcher.ts +++ b/apps/sim/executor/utils/resolved-secret-matcher.ts @@ -293,7 +293,8 @@ export function sanitizeResolvedSecretString( value: string, matcher: ResolvedSecretMatcher, maxBytes = MAX_INLINE_MATERIALIZATION_BYTES, - onMatch?: (plaintext: string) => void + onMatch?: (plaintext: string) => void, + options: { preserveLineBreaks?: boolean } = {} ): string { /** * A substitution can leave a literal the previous pass could not act on: an empty replacement can @@ -302,12 +303,24 @@ export function sanitizeResolvedSecretString( * pathological chain from looping, and the throw past it stays the fail-closed backstop callers * already handle by dropping the value. */ - let sanitized = substituteResolvedSecrets(value, matcher, maxBytes, onMatch) + let sanitized = substituteResolvedSecrets( + value, + matcher, + maxBytes, + onMatch, + options.preserveLineBreaks + ) for (let pass = 1; containsResolvedSecret(sanitized, matcher); pass += 1) { if (pass >= MAX_SETTLE_PASSES) { throw new ResolvedSecretMatcherError('Sanitized content still contains an active secret') } - sanitized = substituteResolvedSecrets(sanitized, matcher, maxBytes, onMatch) + sanitized = substituteResolvedSecrets( + sanitized, + matcher, + maxBytes, + onMatch, + options.preserveLineBreaks + ) } return sanitized } @@ -316,7 +329,8 @@ function substituteResolvedSecrets( value: string, matcher: ResolvedSecretMatcher, maxBytes: number, - onMatch?: (plaintext: string) => void + onMatch?: (plaintext: string) => void, + preserveLineBreaks = false ): string { if (maxBytes < 0) { throw new ResolvedSecretMatcherError('Sanitized secret-bearing string exceeds the size limit') @@ -356,7 +370,21 @@ function substituteResolvedSecrets( const slot = emitCursor % windowSize if (slotStarts[slot] === emitCursor && slotReplacements[slot] !== undefined) { append(value.slice(literalStart, emitCursor)) - append(slotReplacements[slot] ?? '') + const replacement = slotReplacements[slot] ?? '' + if (preserveLineBreaks) { + const lineReplacement = replacement.replace(/[\r\n]/g, ' ') || ' ' + const matched = value.slice(emitCursor, slotEnds[slot]) + let position = 0 + while (position < matched.length) { + append(lineReplacement) + const newline = matched.indexOf('\n', position) + if (newline < 0) break + append(newline > position && matched[newline - 1] === '\r' ? '\r\n' : '\n') + position = newline + 1 + } + } else { + append(replacement) + } emitCursor = slotEnds[slot] literalStart = emitCursor } else { diff --git a/apps/sim/executor/utils/resolved-secret-trace-registry.ts b/apps/sim/executor/utils/resolved-secret-trace-registry.ts index e9158202c07..d5408ec0d88 100644 --- a/apps/sim/executor/utils/resolved-secret-trace-registry.ts +++ b/apps/sim/executor/utils/resolved-secret-trace-registry.ts @@ -1731,6 +1731,41 @@ export class ResolvedSecretTraceRegistry { return importedAll } + /** Imports one item's trusted evidence atomically, leaving this registry unchanged on refusal. */ + async tryImportProvenance( + provenances: Iterable, + options: ImportResolvedSecretTraceProvenanceOptions & { value?: unknown } + ): Promise { + const staged = new ResolvedSecretTraceRegistry([], this.scope, { staged: true }) + for (const provenance of provenances) { + if (!(await staged.importProvenance(provenance, options)) || !staged.isComplete()) + return false + } + + let entries = [...staged.activeEntries.values()] + if (options.value !== undefined) { + const filtered = staged.exportProvenanceForValue(options.value) + if (!filtered.complete) return false + const selected = new Set( + filtered.entries.map((entry) => + activeEntryKey({ + name: entry.name ?? '', + plaintext: '', + encryptedValue: entry.encryptedValue, + anonymous: entry.name === undefined, + }) + ) + ) + entries = entries.filter((entry) => selected.has(activeEntryKey(entry))) + } + + /** No await separates the capacity check from activation, including across concurrent calls. */ + if (this.isPermanentlyIncomplete() || this.activeEntryBytesAfter(entries) === undefined) + return false + for (const entry of entries) this.addActiveEntry(entry, { propagated: true }) + return true + } + /** * Imports only provenance whose decrypted plaintext occurs in the exact value crossing this * boundary. Staging in a scoped temporary registry prevents a valid but overbroad envelope from @@ -2578,6 +2613,22 @@ export class ResolvedSecretTraceRegistry { if (copied) target.inheritIncompletenessReasonsFrom(this) } + private activeEntryBytesAfter(entries: Iterable): number | undefined { + let count = this.activeEntries.size + let bytes = this.activeProvenanceEntryBytes + for (const entry of entries) { + if (this.activeEntries.has(activeEntryKey(entry))) continue + bytes += (count === 0 ? 0 : 1) + serializedProvenanceEntryByteSize(toProvenanceEntry(entry)) + count += 1 + if ( + count > MAX_PROVENANCE_ENTRIES || + this.completeProvenanceEnvelopeBytes + bytes > MAX_SERIALIZED_PROVENANCE_BYTES + ) + return undefined + } + return bytes + } + private addActiveEntry(entry: ActiveSecretEntry, options: { propagated?: boolean } = {}): void { const key = activeEntryKey(entry) const existing = this.activeEntries.get(key) @@ -2596,22 +2647,14 @@ export class ResolvedSecretTraceRegistry { return } - const entryBytes = serializedProvenanceEntryByteSize(toProvenanceEntry(entry)) - const separatorBytes = this.activeEntries.size === 0 ? 0 : 1 - if ( - this.activeEntries.size >= MAX_PROVENANCE_ENTRIES || - this.completeProvenanceEnvelopeBytes + - this.activeProvenanceEntryBytes + - separatorBytes + - entryBytes > - MAX_SERIALIZED_PROVENANCE_BYTES - ) { + const nextEntryBytes = this.activeEntryBytesAfter([entry]) + if (nextEntryBytes === undefined) { this.markIncomplete('provenance-capacity-exceeded') return } this.activeEntries.set(key, entry) if (options.propagated) this.propagatedEntryKeys.add(key) - this.activeProvenanceEntryBytes += separatorBytes + entryBytes + this.activeProvenanceEntryBytes = nextEntryBytes this.modelEgressRevision += 1 } diff --git a/apps/sim/lib/api-key/application/operations.ts b/apps/sim/lib/api-key/application/operations.ts index 52df9771b9c..f71c81cf02c 100644 --- a/apps/sim/lib/api-key/application/operations.ts +++ b/apps/sim/lib/api-key/application/operations.ts @@ -33,6 +33,7 @@ export const apiKeyOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'api_keys.manage', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), @@ -45,6 +46,7 @@ export const byokKeyOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'none', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }), @@ -54,12 +56,14 @@ export const byokKeyOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'none', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }), // permission-group-exempt: BYOK is its own entitlement-gated section, and api_keys.manage names the Sim API Keys tab instead listOrganization: defineOrganizationByokOperation({ id: 'byok_keys.organization.list', + contentDelivery: 'none', capability: 'none', authority: 'organization_admin', minimumRole: 'admin', @@ -84,6 +88,7 @@ export const byokKeyOperations = { // permission-group-exempt: BYOK is its own entitlement-gated section, and api_keys.manage names the Sim API Keys tab instead deleteOrganization: defineOrganizationByokOperation({ id: 'byok_keys.organization.delete', + contentDelivery: 'none', capability: 'none', authority: 'organization_admin', minimumRole: 'admin', @@ -100,6 +105,7 @@ export const byokKeyOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'none', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }), diff --git a/apps/sim/lib/api-key/application/workspace-api-keys.ts b/apps/sim/lib/api-key/application/workspace-api-keys.ts index 9da3e3188b5..7d30ab9c4fb 100644 --- a/apps/sim/lib/api-key/application/workspace-api-keys.ts +++ b/apps/sim/lib/api-key/application/workspace-api-keys.ts @@ -31,12 +31,14 @@ export const workspaceApiKeyOperations = { id: 'api_keys.workspace.list', minimumRole: 'read', capability: 'api_keys.manage', + contentDelivery: 'none', }), rename: defineWorkspaceOperation({ ...principalPolicy, id: 'api_keys.workspace.rename', minimumRole: 'admin', capability: 'api_keys.manage', + contentDelivery: 'none', }), /** permission-group-exempt: admins must retain the ability to revoke exposed credentials. */ revoke: defineWorkspaceOperation({ @@ -44,6 +46,7 @@ export const workspaceApiKeyOperations = { id: 'api_keys.workspace.revoke', minimumRole: 'admin', capability: 'none', + contentDelivery: 'none', }), } as const const authorizationOptions = { delegation: { audience: 'sim:settings', isWithinScope: () => true } } diff --git a/apps/sim/lib/api/server/routes/copilot-content-routes.json b/apps/sim/lib/api/server/routes/copilot-content-routes.json new file mode 100644 index 00000000000..59666a8fdb6 --- /dev/null +++ b/apps/sim/lib/api/server/routes/copilot-content-routes.json @@ -0,0 +1,162 @@ +[ + { + "method": "GET", + "path": "/api/v2/files/[fileId]", + "operation": "files.download" + }, + { + "method": "PATCH", + "path": "/api/v2/files/[fileId]/content", + "operation": "files.update_content" + }, + { + "method": "PUT", + "path": "/api/v2/files/[fileId]/content", + "operation": "files.update_content" + }, + { + "method": "GET", + "path": "/api/v2/files/[fileId]/share", + "operation": "files.share.read" + }, + { + "method": "PATCH", + "path": "/api/v2/files/[fileId]/share", + "operation": "files.share.update" + }, + { + "method": "GET", + "path": "/api/v2/files/[fileId]/text", + "operation": "files.read_content" + }, + { + "method": "GET", + "path": "/api/v2/files/[fileId]/versions/[version]/text", + "operation": "files.versions.read_content" + }, + { + "method": "GET", + "path": "/api/v2/files/bulk-download", + "operation": "files.download" + }, + { + "method": "GET", + "path": "/api/v2/files/search", + "operation": "files.search_content" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/connectors/[connectorId]/documents", + "operation": "knowledge.connectors.documents.list" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents", + "operation": "knowledge.documents.list" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]", + "operation": "knowledge.documents.read" + }, + { + "method": "PATCH", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]", + "operation": "knowledge.documents.update" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]/chunks", + "operation": "knowledge.chunks.list" + }, + { + "method": "POST", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]/chunks", + "operation": "knowledge.chunks.create" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]/chunks/[chunkId]", + "operation": "knowledge.chunks.read" + }, + { + "method": "PATCH", + "path": "/api/v2/knowledge/[knowledgeBaseId]/documents/[documentId]/chunks/[chunkId]", + "operation": "knowledge.chunks.update" + }, + { + "method": "GET", + "path": "/api/v2/knowledge/[knowledgeBaseId]/export", + "operation": "knowledge.export" + }, + { + "method": "POST", + "path": "/api/v2/knowledge/search", + "operation": "knowledge.search" + }, + { + "method": "POST", + "path": "/api/v2/tables/[tableId]/query", + "operation": "tables.rows.query" + }, + { + "method": "POST", + "path": "/api/v2/tables/[tableId]/query/count", + "operation": "tables.rows.query" + }, + { + "method": "GET", + "path": "/api/v2/tables/[tableId]/rows", + "operation": "tables.rows.list" + }, + { + "method": "POST", + "path": "/api/v2/tables/[tableId]/rows", + "operation": "tables.rows.create" + }, + { + "method": "GET", + "path": "/api/v2/tables/[tableId]/rows/[rowId]", + "operation": "tables.rows.read" + }, + { + "method": "PATCH", + "path": "/api/v2/tables/[tableId]/rows/[rowId]", + "operation": "tables.rows.update" + }, + { + "method": "GET", + "path": "/api/v2/tables/[tableId]/rows/[rowId]/enrichment/[groupId]", + "operation": "tables.rows.read" + }, + { + "method": "POST", + "path": "/api/v2/tables/[tableId]/rows/upsert", + "operation": "tables.rows.upsert" + }, + { + "method": "POST", + "path": "/api/v2/tools/[toolId]/execute", + "operation": "tools.execute" + }, + { + "method": "POST", + "path": "/api/v2/workflows/[workflowId]/execute", + "operation": "workflows.execute" + }, + { + "method": "GET", + "path": "/api/v2/workflows/[workflowId]/runs/[runId]", + "operation": "workflows.runs.read" + }, + { + "method": "GET", + "path": "/api/v2/workflows/[workflowId]/runs/[runId]/files/[fileId]", + "operation": "workflows.download_run_file" + }, + { + "method": "POST", + "path": "/api/v2/workflows/[workflowId]/runs/[runId]/resume", + "operation": "workflows.runs.resume" + } +] diff --git a/apps/sim/lib/api/server/routes/copilot-refused-routes.json b/apps/sim/lib/api/server/routes/copilot-refused-routes.json index eacd2e4df6b..2cc89d250fa 100644 --- a/apps/sim/lib/api/server/routes/copilot-refused-routes.json +++ b/apps/sim/lib/api/server/routes/copilot-refused-routes.json @@ -229,6 +229,11 @@ "path": "/api/v2/organizations/[organizationId]/workspaces", "operation": "organizations.workspaces.list" }, + { + "method": "GET", + "path": "/api/v2/tables/[tableId]/exports/[exportId]/download", + "operation": "tables.exports.download" + }, { "method": "GET", "path": "/api/v2/workspaces", diff --git a/apps/sim/lib/api/server/routes/copilot-request.test.ts b/apps/sim/lib/api/server/routes/copilot-request.test.ts index ae34e4a1e31..2212c1d8387 100644 --- a/apps/sim/lib/api/server/routes/copilot-request.test.ts +++ b/apps/sim/lib/api/server/routes/copilot-request.test.ts @@ -36,6 +36,7 @@ const policy = { const operation = defineWorkspaceOperation({ id: 'widgets.read', + contentDelivery: 'none', capability: 'none', minimumRole: 'read', workspaceApiKey: 'deny', diff --git a/apps/sim/lib/api/server/routes/copilot-request.ts b/apps/sim/lib/api/server/routes/copilot-request.ts index 15e841b3e2c..0cb4bc3052a 100644 --- a/apps/sim/lib/api/server/routes/copilot-request.ts +++ b/apps/sim/lib/api/server/routes/copilot-request.ts @@ -1,4 +1,5 @@ import type { DelegatedPrincipal } from '@sim/auth/principal' +import { noteContentDeliveryOperation } from '@/lib/core/application' import { markCopilotWorkspaceInvocation } from '@/lib/core/application/copilot-workspace-invocation' import type { ApplicationOperation, OperationUseCase } from '@/lib/core/application/operation' import { @@ -48,5 +49,6 @@ export function copilotRequestPrincipal( if (!audience) return undefined const principal = createCopilotChatPrincipal(invocation, audience) markCopilotWorkspaceInvocation(principal) + noteContentDeliveryOperation(operation, principal) return principal } diff --git a/apps/sim/lib/api/server/routes/copilot-route-inventory.test.ts b/apps/sim/lib/api/server/routes/copilot-route-inventory.test.ts index 0d02d41f43a..b29ca9fa233 100644 --- a/apps/sim/lib/api/server/routes/copilot-route-inventory.test.ts +++ b/apps/sim/lib/api/server/routes/copilot-route-inventory.test.ts @@ -11,8 +11,8 @@ const inventory = vi.hoisted( [] as Array<{ method: string path: string - operation: ApplicationOperation - useCase: CopilotRouteUseCase + operation: RouteOperation + useCase?: CopilotRouteUseCase }> ) const built = vi.hoisted(() => new WeakSet()) @@ -20,8 +20,8 @@ const builder = vi.hoisted( () => (options: { contract: { method: string; path: string } - operation: ApplicationOperation - useCase: CopilotRouteUseCase + operation: RouteOperation + useCase?: CopilotRouteUseCase }) => { inventory.push({ method: options.contract.method, @@ -40,7 +40,7 @@ const recordAdmission = vi.hoisted( () => async ( _request: Request, - operation: ApplicationOperation, + operation: RouteOperation, _auth: unknown, _rateLimit: unknown, useCase?: CopilotRouteUseCase @@ -71,6 +71,10 @@ import { } from '@/lib/api/server/routes/copilot-request' import { V2_ROUTES } from '@/lib/api/server/routes/v2-route-table.generated' import type { ApplicationOperation } from '@/lib/core/application/operation' +import type { WorkspaceOperation } from '@/lib/core/application/workspace-operation' + +type RouteOperation = ApplicationOperation & + Pick /** * Routes Mothership refuses, judged by the rule admission itself runs. The CLI command @@ -80,8 +84,25 @@ import type { ApplicationOperation } from '@/lib/core/application/operation' */ const REFUSED_ROUTES_FILE = './copilot-refused-routes.json' +/** + * Routes Chat may call that return content from a provenance-bearing store. Each is gated per + * item, and `content-delivery.integration.ts` drives every one through Chat's real transports: a + * route added here without a case there fails that suite. Refresh with `vitest run -u`. + */ +const CONTENT_ROUTES_FILE = './copilot-content-routes.json' + const METHODS = ['GET', 'POST', 'PUT', 'PATCH', 'DELETE'] as const +/** Whether authorization lets a Chat principal through, not only whether the route mints one. */ +function admitsCopilotPrincipal(operation: RouteOperation): boolean { + const kinds: readonly string[] = operation.principalKinds + const services: readonly string[] | undefined = operation.delegatedServices + return ( + (kinds.includes('delegated') || kinds.includes('organization_delegated')) && + (!services || services.includes('copilot')) + ) +} + /** * A raw route (chat, workflow execute and resume) admits inside its handler, so it is * invoked once and its admission call recorded, which ends the request before any work. @@ -112,4 +133,22 @@ it('inventories private operation admission without executing use cases', async .map(({ method, path, operation }) => ({ method, path, operation: operation.id })) await expect(`${JSON.stringify(refused, null, 2)}\n`).toMatchFileSnapshot(REFUSED_ROUTES_FILE) expect(audiences.filter(Boolean).every((audience) => audience?.startsWith('sim:'))).toBe(true) + + /** + * Every route Chat may call says whether it returns stored content, so no route reaches Chat's + * model by being forgotten: an undeclared one is listed here, and definition refuses it too. + */ + const admitted = inventory.filter( + (route, index) => audiences[index] && admitsCopilotPrincipal(route.operation) + ) + expect( + admitted + .filter((route) => route.operation.contentDelivery === undefined) + .map(({ method, path, operation }) => `${method} ${path} (${operation.id})`) + ).toEqual([]) + const sourced = admitted + .filter((route) => route.operation.contentDelivery === 'sourced') + .map(({ method, path, operation }) => ({ method, path, operation: operation.id })) + .sort((a, b) => `${a.path} ${a.method}`.localeCompare(`${b.path} ${b.method}`)) + await expect(`${JSON.stringify(sourced, null, 2)}\n`).toMatchFileSnapshot(CONTENT_ROUTES_FILE) }, 60000) diff --git a/apps/sim/lib/audit-logs/application/operations.ts b/apps/sim/lib/audit-logs/application/operations.ts index 2f56b86ce32..2aa79688df7 100644 --- a/apps/sim/lib/audit-logs/application/operations.ts +++ b/apps/sim/lib/audit-logs/application/operations.ts @@ -45,6 +45,7 @@ export const auditLogOperations = { // permission-group-exempt: the organization audit trail is authorized by organization admin or owner role, a scope no workspace-shaped permission group can name list: defineAuditLogOperation({ id: 'audit_logs.list', + contentDelivery: 'none', oauthScope: 'api:read', capability: 'none', authority: 'organization_admin', @@ -55,6 +56,7 @@ export const auditLogOperations = { // permission-group-exempt: same organization-admin authority as the list it expands; no group key names the audit trail readDetail: defineAuditLogOperation({ id: 'audit_logs.read_detail', + contentDelivery: 'none', oauthScope: 'api:read', capability: 'none', authority: 'organization_admin', @@ -71,6 +73,7 @@ export const copilotAuditLogOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'copilot.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), @@ -79,6 +82,7 @@ export const copilotAuditLogOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'copilot.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), diff --git a/apps/sim/lib/benchmarks/application/access.ts b/apps/sim/lib/benchmarks/application/access.ts index 8a1f529a1da..2e619d5388b 100644 --- a/apps/sim/lib/benchmarks/application/access.ts +++ b/apps/sim/lib/benchmarks/application/access.ts @@ -76,6 +76,7 @@ export function defineAuthorizedBenchmarkUseCase< const targetOperation = defineOrganizationOperation({ id: 'benchmarks.target.authorize', + contentDelivery: 'none', minimumRole: 'member', capability: 'copilot.use', principalKinds: ['organization_delegated'], diff --git a/apps/sim/lib/benchmarks/application/operations.ts b/apps/sim/lib/benchmarks/application/operations.ts index 2225f135b7c..2546e6faa7a 100644 --- a/apps/sim/lib/benchmarks/application/operations.ts +++ b/apps/sim/lib/benchmarks/application/operations.ts @@ -102,6 +102,7 @@ export const benchmarkOperations = { export const benchmarkSourceOperation = defineWorkspaceOperation({ id: 'benchmarks.source.read', + contentDelivery: 'none', minimumRole: 'read', workspaceApiKey: 'deny', principalKinds: ['delegated'], diff --git a/apps/sim/lib/billing/application/member-usage-limits/operations.ts b/apps/sim/lib/billing/application/member-usage-limits/operations.ts index b99a9f2e6fc..6f8ee79ef2f 100644 --- a/apps/sim/lib/billing/application/member-usage-limits/operations.ts +++ b/apps/sim/lib/billing/application/member-usage-limits/operations.ts @@ -6,6 +6,7 @@ export const memberUsageLimitOperations = { */ read: defineOrganizationOperation({ id: 'organization_member_usage_limits.read', + contentDelivery: 'none', minimumRole: 'admin', capability: 'none', principalKinds: ['session', 'personal_api_key', 'oauth_access_token', 'organization_delegated'], @@ -18,6 +19,7 @@ export const memberUsageLimitOperations = { */ update: defineOrganizationOperation({ id: 'organization_member_usage_limits.update', + contentDelivery: 'none', minimumRole: 'admin', capability: 'none', principalKinds: ['session', 'personal_api_key', 'oauth_access_token', 'organization_delegated'], diff --git a/apps/sim/lib/billing/application/operations.ts b/apps/sim/lib/billing/application/operations.ts index 9f675b07f32..259c70610df 100644 --- a/apps/sim/lib/billing/application/operations.ts +++ b/apps/sim/lib/billing/application/operations.ts @@ -40,6 +40,7 @@ export const billingOperations = { // permission-group-exempt: a personal account reading its own plan and balance; permission groups scope a workspace, not the billing account that owns it readStatus: defineBillingReadOperation({ id: 'billing.status.read', + contentDelivery: 'none', oauthScope: 'api:read', capability: 'none', accountScope: 'personal_self', @@ -50,6 +51,7 @@ export const billingOperations = { // permission-group-exempt: the same personal billing account reading its own usage records; no group key names it listLogs: defineBillingReadOperation({ id: 'billing.logs.list', + contentDelivery: 'none', oauthScope: 'api:read', capability: 'none', accountScope: 'personal_self', @@ -66,6 +68,7 @@ export const copilotBillingOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'copilot.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), @@ -74,6 +77,7 @@ export const copilotBillingOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'copilot.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), diff --git a/apps/sim/lib/billing/application/organization-usage/operations.ts b/apps/sim/lib/billing/application/organization-usage/operations.ts index 5d3ca5f4600..cb089dd3bb1 100644 --- a/apps/sim/lib/billing/application/organization-usage/operations.ts +++ b/apps/sim/lib/billing/application/organization-usage/operations.ts @@ -44,6 +44,7 @@ export const organizationUsageOperations = { */ readActivitySummary: defineOrganizationOperation({ id: 'organization_usage.activity.summary.read', + contentDelivery: 'none', capability: 'none', ...BASE, }), @@ -52,6 +53,7 @@ export const organizationUsageOperations = { */ readActivityBreakdown: defineOrganizationOperation({ id: 'organization_usage.activity.breakdown.read', + contentDelivery: 'none', capability: 'none', ...BASE, }), @@ -60,6 +62,7 @@ export const organizationUsageOperations = { */ readOverview: defineOrganizationOperation({ id: 'organization_usage.overview.read', + contentDelivery: 'none', capability: 'none', ...BASE, }), @@ -68,6 +71,7 @@ export const organizationUsageOperations = { */ readSummary: defineOrganizationOperation({ id: 'organization_usage.summary.read', + contentDelivery: 'none', capability: 'none', ...PUBLIC_READ, }), @@ -76,6 +80,7 @@ export const organizationUsageOperations = { */ readBreakdown: defineOrganizationOperation({ id: 'organization_usage.breakdown.read', + contentDelivery: 'none', capability: 'none', ...PUBLIC_READ, }), @@ -84,6 +89,7 @@ export const organizationUsageOperations = { */ listEvents: defineOrganizationOperation({ id: 'organization_usage.events.list', + contentDelivery: 'none', capability: 'none', ...PUBLIC_READ, }), @@ -92,6 +98,7 @@ export const organizationUsageOperations = { */ exportEvents: defineOrganizationOperation({ id: 'organization_usage.events.export', + contentDelivery: 'none', capability: 'none', ...BASE, }), diff --git a/apps/sim/lib/catalog/application/operations.ts b/apps/sim/lib/catalog/application/operations.ts index 4c13c97beca..62713f5e349 100644 --- a/apps/sim/lib/catalog/application/operations.ts +++ b/apps/sim/lib/catalog/application/operations.ts @@ -27,6 +27,7 @@ export const catalogOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -43,6 +44,7 @@ export const catalogOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -59,6 +61,7 @@ export const catalogOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -75,6 +78,7 @@ export const catalogOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -95,6 +99,7 @@ export const catalogOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', diff --git a/apps/sim/lib/changelog/application/availability.ts b/apps/sim/lib/changelog/application/availability.ts index 211f65dd7f0..69b307ab711 100644 --- a/apps/sim/lib/changelog/application/availability.ts +++ b/apps/sim/lib/changelog/application/availability.ts @@ -9,6 +9,7 @@ const availabilityOperation = defineWorkspaceOperation({ oauthScope: 'api:read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', diff --git a/apps/sim/lib/changelog/application/operations.ts b/apps/sim/lib/changelog/application/operations.ts index efa05caac93..a67e795079d 100644 --- a/apps/sim/lib/changelog/application/operations.ts +++ b/apps/sim/lib/changelog/application/operations.ts @@ -12,6 +12,7 @@ export const changelogOperations = { ...policy, oauthScope: 'api:read', capability: 'files.use', + contentDelivery: 'none', id: 'changelog.read', minimumRole: 'read', }), @@ -19,6 +20,7 @@ export const changelogOperations = { ...policy, oauthScope: 'api:write', capability: 'files.use', + contentDelivery: 'none', id: 'changelog.publish', minimumRole: 'write', }), @@ -26,6 +28,7 @@ export const changelogOperations = { ...policy, oauthScope: 'api:write', capability: 'files.use', + contentDelivery: 'none', id: 'changelog.update', minimumRole: 'write', }), diff --git a/apps/sim/lib/chat-deployments/application/operations.ts b/apps/sim/lib/chat-deployments/application/operations.ts index 4673eb3023d..f689ad160e5 100644 --- a/apps/sim/lib/chat-deployments/application/operations.ts +++ b/apps/sim/lib/chat-deployments/application/operations.ts @@ -54,6 +54,7 @@ export const chatDeploymentOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'deploy.chat', + contentDelivery: 'none', ...CHAT_DEPLOYMENT_LIST_POLICY, }), replace: defineWorkspaceOperation({ @@ -62,6 +63,7 @@ export const chatDeploymentOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.chat', + contentDelivery: 'none', ...CHAT_DEPLOYMENT_ADMIN_POLICY, }), read: defineWorkspaceOperation({ @@ -70,6 +72,7 @@ export const chatDeploymentOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.chat', + contentDelivery: 'none', ...CHAT_DEPLOYMENT_ADMIN_POLICY, }), update: defineWorkspaceOperation({ @@ -78,6 +81,7 @@ export const chatDeploymentOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.chat', + contentDelivery: 'none', ...CHAT_DEPLOYMENT_ADMIN_POLICY, }), delete: defineWorkspaceOperation({ @@ -86,6 +90,7 @@ export const chatDeploymentOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.chat', + contentDelivery: 'none', ...CHAT_DEPLOYMENT_ADMIN_POLICY, }), } as const diff --git a/apps/sim/lib/core/application/authorized-workspace-use-case.ts b/apps/sim/lib/core/application/authorized-workspace-use-case.ts index d0acf45beee..233c51e9d74 100644 --- a/apps/sim/lib/core/application/authorized-workspace-use-case.ts +++ b/apps/sim/lib/core/application/authorized-workspace-use-case.ts @@ -1,6 +1,7 @@ import { type AuditActionType, type AuditResourceTypeValue, recordAudit } from '@sim/audit' import type { Principal, PrincipalAuditAttribution } from '@sim/auth/principal' import { resolvePrincipalAuditAttribution } from '@sim/auth/principal' +import { runDeclaredContentOperation } from '@/lib/core/application/content-delivery' import type { ApplicationOperation, OperationUseCase } from '@/lib/core/application/operation' import { authorizeWorkspaceOperation, @@ -214,27 +215,30 @@ export function defineAuthorizedWorkspaceUseCase< async execute(args) { const executionContext = await authorizePhase(args) const { principal, context, request } = executionContext - return runWithOutboundOrganization(context.workspaceOrganizationId, async () => { - const result = await withinAuthorizedWorkspaceOperation(() => - definition.execute(executionContext) - ) - const resultContext = { ...executionContext, result } - const projectedAudit = definition.projectAudit?.(resultContext) - if (projectedAudit !== undefined) { - const auditEntries = Array.isArray(projectedAudit) ? projectedAudit : [projectedAudit] - if (auditEntries.length > 0) { - recordProjectedUseCaseAuditEntries( - definition.operation, - context.workspaceId, - principal, - request, - auditEntries - ) + return runWithOutboundOrganization(context.workspaceOrganizationId, () => + runDeclaredContentOperation( + definition.operation, + principal, + () => withinAuthorizedWorkspaceOperation(() => definition.execute(executionContext)), + async (result) => { + const resultContext = { ...executionContext, result } + const projectedAudit = definition.projectAudit?.(resultContext) + if (projectedAudit !== undefined) { + const auditEntries = Array.isArray(projectedAudit) ? projectedAudit : [projectedAudit] + if (auditEntries.length > 0) { + recordProjectedUseCaseAuditEntries( + definition.operation, + context.workspaceId, + principal, + request, + auditEntries + ) + } + } + await definition.afterSuccess?.(resultContext) } - } - await definition.afterSuccess?.(resultContext) - return result - }) + ) + ) }, } } diff --git a/apps/sim/lib/core/application/content-delivery.test.ts b/apps/sim/lib/core/application/content-delivery.test.ts new file mode 100644 index 00000000000..c0db1b09b00 --- /dev/null +++ b/apps/sim/lib/core/application/content-delivery.test.ts @@ -0,0 +1,499 @@ +/** + * @vitest-environment node + */ +import { AsyncLocalStorage } from 'node:async_hooks' +import type { DelegatedServiceId, Principal } from '@sim/auth/principal' +import { describe, expect, it, vi } from 'vitest' +import { + admitContent, + admitWrittenContent, + type ContentDeliveryGate, + ContentDeliveryUnaccountedError, + ContentDeliveryUnobservedError, + ContentRefusedError, + type ContentSource, + mustClassifyContent, + noteContentDeliveryOperation, + observeContentDelivery, + observeContentOutsideModel, + requireAdmittedContent, + runDeclaredContentOperation, + trackContentDelivery, +} from '@/lib/core/application/content-delivery' +import { defineWorkspaceOperation } from '@/lib/core/application/workspace-operation' +import { PROVENANCE_MAX_ENTRIES } from '@/lib/execution/provenance-limits' + +function delegated(serviceId: DelegatedServiceId): Principal { + const now = Date.now() + return { + kind: 'delegated', + serviceId, + subjectUserId: 'user-1', + workspaceId: 'workspace-1', + delegationId: 'delegation-1', + audience: 'sim:test', + issuedAt: new Date(now), + expiresAt: new Date(now + 60_000), + } +} + +const CHAT = delegated('copilot') +const EXECUTOR = delegated('executor') +const SESSION: Principal = { kind: 'session', userId: 'user-1', sessionId: 'session-1' } + +function operation(contentDelivery: 'sourced' | 'none') { + return defineWorkspaceOperation({ + id: `test.${contentDelivery}`, + contentDelivery, + minimumRole: 'read', + workspaceApiKey: 'deny', + principalKinds: ['delegated'], + delegatedServices: ['copilot'], + capability: 'none', + }) +} + +const SOURCED = operation('sourced') +const NONE = operation('none') + +function source(id: string, status: 'clean' | 'unknown' = 'clean'): ContentSource { + return { + kind: 'workspace-file', + id, + provenance: status === 'clean' ? { status: 'exact', entries: [] } : { status: 'unknown' }, + view: 'complete', + } +} + +/** Admits clean items and refuses unknown ones, recording every batch it was asked about. */ +function recordingGate() { + const batches: string[][] = [] + const gate: ContentDeliveryGate = { + async admit(sources) { + batches.push(sources.map((item) => item.id)) + return sources.map((item) => item.provenance.status !== 'unknown') + }, + } + return { gate, batches } +} + +describe('mustClassifyContent', () => { + it('refuses Chat with no gate before anything loads', () => { + expect(() => mustClassifyContent(CHAT)).toThrow(ContentDeliveryUnobservedError) + }) + + it('is true for Chat under a gate and false for every other caller, gate or not', () => { + const { gate } = recordingGate() + expect(observeContentDelivery(gate, () => mustClassifyContent(CHAT))).toBe(true) + expect(mustClassifyContent(EXECUTOR)).toBe(false) + expect(mustClassifyContent(SESSION)).toBe(false) + expect(observeContentDelivery(gate, () => mustClassifyContent(SESSION))).toBe(false) + }) +}) + +describe('admitContent', () => { + it('decides each item on its own for Chat', async () => { + const { gate, batches } = recordingGate() + const items = [source('clean'), source('tainted', 'unknown'), source('also-clean')] + const admission = await observeContentDelivery(gate, () => admitContent(CHAT, items)) + expect(items.map((item) => admission.admits(item))).toEqual([true, false, true]) + expect(admission.refused.map((item) => item.id)).toEqual(['tainted']) + expect(batches).toEqual([['clean', 'tainted', 'also-clean']]) + }) + + it('admits every other caller unread, whatever the items say', async () => { + const { gate, batches } = recordingGate() + const items = [source('tainted', 'unknown')] + for (const principal of [EXECUTOR, SESSION]) { + const admission = await observeContentDelivery(gate, () => admitContent(principal, items)) + expect(admission.admits(items[0])).toBe(true) + } + expect(batches).toEqual([]) + }) + + it('only accounts for an empty result, even for Chat with no gate', async () => { + const record = await trackContentDelivery(async () => { + noteContentDeliveryOperation(SOURCED, CHAT) + return admitContent(CHAT, []) + }) + expect(record.result.refused).toEqual([]) + expect(record.unaccounted).toBe(false) + }) + + it('refuses Chat items with no gate installed', async () => { + await expect(admitContent(CHAT, [source('clean')])).rejects.toThrow( + ContentDeliveryUnobservedError + ) + }) + + it('refuses items classified inside an operation declared to return none', async () => { + const { gate } = recordingGate() + await expect( + observeContentDelivery(gate, () => + runDeclaredContentOperation(NONE, CHAT, () => admitContent(CHAT, [source('clean')])) + ) + ).rejects.toMatchObject({ name: 'ContentDeliveryUndeclaredError' }) + }) +}) + +describe('requireAdmittedContent', () => { + it('passes an admitted item and refuses a refused one with a 403 naming no content', async () => { + const { gate } = recordingGate() + await expect( + observeContentDelivery(gate, () => requireAdmittedContent(CHAT, source('clean'))) + ).resolves.toBeUndefined() + const refusal = await observeContentDelivery(gate, () => + requireAdmittedContent(CHAT, source('tainted', 'unknown')).catch((error: unknown) => error) + ) + expect(refusal).toBeInstanceOf(ContentRefusedError) + expect(refusal).toMatchObject({ + code: 'forbidden', + message: expect.not.stringContaining('tainted'), + }) + }) +}) + +describe('runDeclaredContentOperation', () => { + it('throws for a sourced operation that never accounted, after its effects completed', async () => { + const { gate } = recordingGate() + const complete = vi.fn(async () => {}) + await expect( + observeContentDelivery(gate, () => + runDeclaredContentOperation(SOURCED, CHAT, async () => 'unclassified', complete) + ) + ).rejects.toThrow(ContentDeliveryUnaccountedError) + expect(complete).toHaveBeenCalledWith('unclassified') + }) + + it('returns a sourced result that accounted, and a none result that had nothing to account', async () => { + const { gate } = recordingGate() + await expect( + observeContentDelivery(gate, () => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitContent(CHAT, [source('clean')]) + return 'classified' + }) + ) + ).resolves.toBe('classified') + await expect(runDeclaredContentOperation(NONE, CHAT, async () => 'metadata')).resolves.toBe( + 'metadata' + ) + }) + + it('runs every other caller unchanged, with no accounting demanded', async () => { + const complete = vi.fn(async () => {}) + await expect( + runDeclaredContentOperation(SOURCED, EXECUTOR, async () => 'run output', complete) + ).resolves.toBe('run output') + expect(complete).toHaveBeenCalledWith('run output') + }) + + it('answers a committed write whose returned item was refused only after its effects', async () => { + const { gate } = recordingGate() + const events: string[] = [] + const refusal = await observeContentDelivery(gate, () => + runDeclaredContentOperation( + SOURCED, + CHAT, + async () => { + const admission = await admitWrittenContent(CHAT, [source('written', 'unknown')]) + events.push(`write returned admitted=${admission.refused.length === 0}`) + return 'saved row' + }, + async () => { + events.push('audit recorded') + } + ).catch((error: unknown) => error) + ) + expect(events).toEqual(['write returned admitted=false', 'audit recorded']) + expect(refusal).toBeInstanceOf(ContentRefusedError) + expect(refusal).toMatchObject({ message: expect.stringContaining('The change was saved') }) + }) + + it.each(['workspace-file', 'tool-result'] as const)( + 'finishes %s effects before reporting a gate failure with its actual completion state', + async (kind) => { + const failure = new Error('history unavailable') + const failing: ContentDeliveryGate = { + async admit() { + throw failure + }, + } + const events: string[] = [] + const outcome = await observeContentDelivery(failing, () => + runDeclaredContentOperation( + SOURCED, + CHAT, + async () => { + const admission = await admitWrittenContent(CHAT, [{ ...source('written'), kind }]) + events.push(`withheld=${admission.refused.length}`) + return 'saved row' + }, + async () => { + events.push('audit recorded') + } + ).catch((error: unknown) => error) + ) + expect(events).toEqual(['withheld=1', 'audit recorded']) + /** Not the gate's error: that would invite a retry of a write that already committed. */ + expect(outcome).toBeInstanceOf(ContentRefusedError) + expect(outcome).toMatchObject({ + message: expect.stringContaining( + kind === 'tool-result' ? 'Tool execution settled' : 'The change was saved' + ), + }) + /** The request's transport still sees the failure, to answer it itself (the proxy's 502). */ + const record = await observeContentDelivery(failing, () => + trackContentDelivery(() => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitWrittenContent(CHAT, [{ ...source('written'), kind }]) + return 'saved row' + }).catch(() => 'answered') + ) + ) + expect(record.gateFailure).toBe(failure) + expect(record.committedWrite).toBe(kind !== 'tool-result') + await expect( + observeContentDelivery(failing, () => + trackContentDelivery(() => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitWrittenContent(CHAT, [{ ...source('written'), kind }]) + }) + ) + ) + ).rejects.toMatchObject({ gateFailure: failure, committedWrite: kind !== 'tool-result' }) + } + ) + + it('does not account for a result whose admission failed and was caught', async () => { + const failing: ContentDeliveryGate = { + async admit() { + throw new Error('gate down') + }, + } + const outcome = await observeContentDelivery(failing, () => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitContent(CHAT, [source('read')]).catch(() => undefined) + return 'content returned anyway' + }).catch((error: unknown) => error) + ) + expect(outcome).toBeInstanceOf(ContentDeliveryUnaccountedError) + }) + + it('does not let a nested use case account for what its enclosing operation returns', async () => { + const admitAll: ContentDeliveryGate = { + async admit(sources) { + return sources.map(() => true) + }, + } + const outcome = await observeContentDelivery(admitAll, () => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitContent(CHAT, [source('nested')]) + return 'nested' + }) + return 'outer content never classified' + }).catch((error: unknown) => error) + ) + expect(outcome).toBeInstanceOf(ContentDeliveryUnaccountedError) + }) + + it('rethrows a gate failure at once when no operation is open to answer with it', async () => { + const failure = new Error('history unavailable') + const failing: ContentDeliveryGate = { + async admit() { + throw failure + }, + } + await expect( + observeContentDelivery(failing, () => admitWrittenContent(CHAT, [source('written')])) + ).rejects.toBe(failure) + }) + + it('lets a written result through when every returned item was admitted', async () => { + const { gate } = recordingGate() + await expect( + observeContentDelivery(gate, () => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitWrittenContent(CHAT, [source('written')]) + return 'saved row' + }) + ) + ).resolves.toBe('saved row') + }) +}) + +describe('trackContentDelivery', () => { + it('reports a sourced operation Chat was admitted to whose use case never accounted', async () => { + const record = await trackContentDelivery(async () => { + noteContentDeliveryOperation(SOURCED, CHAT) + return 'response' + }) + expect(record).toEqual({ + result: 'response', + unaccounted: true, + committedWrite: false, + admitted: [], + }) + }) + + it('does not report a none operation, or a sourced one another caller ran', async () => { + for (const [op, principal] of [ + [NONE, CHAT], + [SOURCED, EXECUTOR], + ] as const) { + const record = await trackContentDelivery(async () => { + noteContentDeliveryOperation(op, principal) + }) + expect(record.unaccounted).toBe(false) + } + }) + + it('collects the provenance of admitted items only, through nested use cases', async () => { + const { gate } = recordingGate() + const entries = { + status: 'exact' as const, + entries: [{ encryptedValue: 'ciphertext', sourceUserId: 'user-1' }], + } + const record = await observeContentDelivery(gate, () => + trackContentDelivery(() => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitContent(CHAT, [ + { ...source('inner'), provenance: entries }, + source('refused', 'unknown'), + ]) + }) + await admitContent(CHAT, [source('outer')]) + }) + ) + ) + expect(record.unaccounted).toBe(false) + expect(record.admitted).toEqual([entries]) + }) + + it('keeps one bounded provenance union when nested deferred pages repeat admitted evidence', async () => { + const gate: ContentDeliveryGate = { + async admit(items) { + return items.map(() => true) + }, + } + const entries = [ + { encryptedValue: 'first-ciphertext', sourceUserId: 'user-1' }, + { encryptedValue: 'second-ciphertext', sourceUserId: 'user-1' }, + ] + const record = await observeContentDelivery(gate, () => + trackContentDelivery(() => + runDeclaredContentOperation(SOURCED, CHAT, async () => { + await admitContent(CHAT, []) + const inRequest = AsyncLocalStorage.snapshot() + return () => + inRequest(() => + trackContentDelivery(async () => { + await admitContent( + CHAT, + entries.map((entry, index) => ({ + ...source(`chunk-${index}`), + provenance: { status: 'exact', entries: [entry] }, + })) + ) + }) + ) + }) + ) + ) + expect(record.admitted).toEqual([]) + for (let page = 0; page < 200; page += 1) { + const nested = await record.result() + expect(nested.admitted).toEqual([{ status: 'exact', entries }]) + } + expect(record.admitted).toEqual([{ status: 'exact', entries }]) + }) + + it('retains unrecorded coverage without retaining every clean or unrecorded source', async () => { + const gate: ContentDeliveryGate = { + async admit(items) { + return items.map(() => true) + }, + } + const record = await observeContentDelivery(gate, () => + trackContentDelivery(async () => { + for (let page = 0; page < 200; page += 1) { + await admitContent(CHAT, [ + source('clean'), + { ...source('legacy'), provenance: { status: 'unrecorded' } }, + ]) + } + }) + ) + expect(record.admitted).toEqual([{ status: 'unrecorded' }]) + }) + + it('uses the existing provenance budget for combined delivered evidence', async () => { + const gate: ContentDeliveryGate = { + async admit(items) { + return items.map(() => true) + }, + } + const entries = Array.from({ length: PROVENANCE_MAX_ENTRIES }, (_, index) => ({ + encryptedValue: `ciphertext-${index}`, + sourceUserId: 'user-1', + })) + const record = await observeContentDelivery(gate, () => + trackContentDelivery(async () => { + await admitContent(CHAT, [{ ...source('first'), provenance: { status: 'exact', entries } }]) + await admitContent(CHAT, [ + { + ...source('overflow'), + provenance: { + status: 'exact', + entries: [{ encryptedValue: 'one-more', sourceUserId: 'user-1' }], + }, + }, + ]) + }) + ) + expect(record.admitted).toEqual([{ status: 'unknown' }]) + }) + + it('keeps a nested request’s accounting from satisfying a sibling that never accounted', async () => { + const { gate } = recordingGate() + const [accounted, unaccounted] = await observeContentDelivery(gate, () => + Promise.all([ + trackContentDelivery(async () => { + noteContentDeliveryOperation(SOURCED, CHAT) + await admitContent(CHAT, [source('clean')]) + }), + trackContentDelivery(async () => { + noteContentDeliveryOperation(SOURCED, CHAT) + }), + ]) + ) + expect(accounted.unaccounted).toBe(false) + expect(unaccounted.unaccounted).toBe(true) + }) +}) + +describe('observeContentDelivery', () => { + it('decides by the innermost gate', async () => { + const outer = recordingGate() + const inner = recordingGate() + await observeContentDelivery(outer.gate, () => + observeContentDelivery(inner.gate, () => admitContent(CHAT, [source('clean')])) + ) + expect(inner.batches).toEqual([['clean']]) + expect(outer.batches).toEqual([]) + }) +}) + +describe('observeContentOutsideModel', () => { + it('admits every item, tainted ones included, inside a gate that would refuse them', async () => { + const { gate, batches } = recordingGate() + const tainted = source('tainted', 'unknown') + const admission = await observeContentDelivery(gate, () => + observeContentOutsideModel(() => admitContent(CHAT, [tainted])) + ) + expect(admission.admits(tainted)).toBe(true) + expect(batches).toEqual([]) + }) +}) diff --git a/apps/sim/lib/core/application/content-delivery.ts b/apps/sim/lib/core/application/content-delivery.ts new file mode 100644 index 00000000000..d07902baf2a --- /dev/null +++ b/apps/sim/lib/core/application/content-delivery.ts @@ -0,0 +1,506 @@ +import { AsyncLocalStorage } from 'node:async_hooks' +import type { Principal } from '@sim/auth/principal' +import type { ApplicationOperation } from '@/lib/core/application/operation' +import { OrchestrationError } from '@/lib/core/orchestration/types' +import type { DurableSecretProvenance } from '@/lib/execution/durable-secret-provenance' +import { mergeDurableSecretProvenanceSources } from '@/lib/execution/provenance-entries' + +/** + * The secret provenance of one stored item: `exact` names every secret its content can hold, + * `unrecorded` is a writer that recorded nothing (audited, otherwise ordinary input), and `unknown` + * is taint or a stale binding, which no policy relaxes. + */ +export type ContentProvenance = DurableSecretProvenance | { status: 'unrecorded' } + +export type ContentSourceKind = + | 'workspace-file' + | 'execution-file' + | 'knowledge-document' + | 'knowledge-chunk' + | 'table-row' + | 'tool-result' + +/** + * How an item's content leaves the use case. `complete` is text in which any of its secrets may + * appear; `derived` is text built from part of it, so only secrets occurring in `value` count; + * `opaque` is bytes no text redaction reaches — binary, base64, archives, images, public links; + * `excerpt` is text already cut before admission, so a secret fragment cannot be redacted. + */ +export type ContentView = 'complete' | 'opaque' | 'excerpt' | { derived: unknown } + +/** One stored item or tool result whose content a use case is about to return. */ +export interface ContentSource { + kind: ContentSourceKind + /** Canonical id, for attribution and per-item exclusion only. */ + id: string + provenance: ContentProvenance + view: ContentView +} + +/** + * Decides, per item, whether content may leave through the surface that installed it. A gate never + * throws for a refusal: refusing one item must not cost the call the items it admits. + */ +export interface ContentDeliveryGate { + admit(sources: readonly ContentSource[]): Promise + /** + * Redacts text before cutting, or returns `undefined` when it cannot be shown. The current + * operation's bounded evidence lets a surface create a redactor only when a cut needs one. + */ + project?( + text: string, + provenance?: ContentProvenance + ): string | undefined | Promise +} + +/** + * Whether a declared operation's result can carry content derived from a provenance-bearing store + * (workspace files and their versions, execution files, knowledge documents and chunks, table + * rows): bytes, text, excerpts, cells, previews, base64, archives, public links to such content, or + * returned matches. `none` is spelled out, as `capability` is, so an operation Chat can reach + * cannot reach production without someone deciding. + */ +export type ContentDeliveryDeclaration = 'sourced' | 'none' + +/** Accounting for one dispatched request or one use-case execution. */ +interface ContentLedger { + parent?: ContentLedger + /** The declaration of the operation this ledger runs, when it runs one. */ + declaration?: ContentDeliveryDeclaration + /** Chat was admitted to a `sourced` operation while this ledger was open. */ + sourcedOperationAdmitted: boolean + /** A use case accounted for the sources it returns, even when it returned none. */ + accounted: boolean + /** Bounded aggregate of admitted provenance, for a surface that persists the delivered bytes. */ + admitted: ContentProvenance[] + /** A committed write's returned item was refused; the operation answers with a refusal. */ + withheldAfterWrite?: ContentSourceKind + /** The gate itself failed while this ledger was open, whatever the route then answered. */ + gateFailure?: unknown +} + +const gates = new AsyncLocalStorage() +const ledgers = new AsyncLocalStorage() + +const MODEL_UNSAFE_CONTENT_MESSAGES: Record = { + 'workspace-file': 'File cannot be sent to a model because its secret provenance is unavailable', + 'execution-file': + 'Run file cannot be sent to a model because its secret provenance is unavailable', + 'knowledge-document': + 'Document cannot be sent to a model because its secret provenance is unavailable', + 'knowledge-chunk': 'Chunk cannot be sent to a model because its secret provenance is unavailable', + 'table-row': 'Row cannot be sent to a model because its secret provenance is unavailable', + 'tool-result': + 'Tool result cannot be sent to a model because its secret provenance is unavailable', +} + +/** + * A gate refused an item. A `403` like any refusal, but its own class, so a caller assembling many + * items can leave this one out without mistaking a permission denial for it. + */ +export class ContentRefusedError extends OrchestrationError { + /** `afterWrite` names a refusal of a write's returned content once the write has committed. */ + constructor(kind: ContentSourceKind, options: { afterWrite?: boolean } = {}) { + super( + 'forbidden', + options.afterWrite + ? kind === 'tool-result' + ? 'Tool execution settled, but its result could not be returned safely. Do not retry automatically.' + : `The change was saved. ${MODEL_UNSAFE_CONTENT_MESSAGES[kind]}, so the result is not shown; do not retry the change.` + : MODEL_UNSAFE_CONTENT_MESSAGES[kind] + ) + this.name = 'ContentRefusedError' + } +} + +/** Chat reached stored content with no gate to classify it. */ +export class ContentDeliveryUnobservedError extends Error { + constructor() { + super('Content provenance is unavailable for this read. Retry the read.') + this.name = 'ContentDeliveryUnobservedError' + } +} + +/** A `sourced` operation completed for Chat without accounting for the content it returned. */ +export class ContentDeliveryUnaccountedError extends Error { + constructor(operationId: string) { + super(`Operation ${operationId} returned content to Chat without classifying it`) + this.name = 'ContentDeliveryUnaccountedError' + } +} + +/** Content was classified for Chat inside an operation declared to return none. */ +class ContentDeliveryUndeclaredError extends Error { + constructor() { + super('Content was delivered to Chat by an operation declared to return none') + this.name = 'ContentDeliveryUndeclaredError' + } +} + +/** Chat's own principals: the in-workspace delegation and the organization chat delegation. */ +export function isCopilotPrincipal(principal: Principal): boolean { + return ( + (principal.kind === 'delegated' || principal.kind === 'organization_delegated') && + principal.serviceId === 'copilot' + ) +} + +/** Installs the gate every item reported inside `execute` is decided by; the innermost gate wins. */ +export function observeContentDelivery(gate: ContentDeliveryGate, execute: () => T): T { + return gates.run(gate, execute) +} + +/** + * Text a use case is about to cut (a line window, a clip), redacted first by the surface's gate. + * Egress redacts whole secrets only, so a cut through one would leave a fragment it cannot + * recognize. Other callers and explicit outside-model reads get the text unchanged. + */ +export async function redactBeforeCutting( + principal: Principal | undefined, + text: string, + kind: ContentSourceKind +): Promise { + if (principal && !isCopilotPrincipal(principal)) return text + const project = gates.getStore()?.project + if (!project) return text + const projected = await project(text, ledgers.getStore()?.admitted[0]) + if (projected === undefined) throw new ContentRefusedError(kind) + return projected +} + +/** Admits every item: see {@link observeContentOutsideModel}. */ +const OUTSIDE_MODEL_GATE: ContentDeliveryGate = { + async admit(sources) { + return sources.map(() => true) + }, +} + +/** + * Reads, for Chat, content that never reaches its model: input to a stored artifact whose + * classification the caller derives from every source it read (a transcoded media file), or the + * user's own view of a file being edited. Items read inside `execute` are admitted unclassified, + * so a tainted source still yields a derived file that carries its taint. Each use names why. + */ +export function observeContentOutsideModel(execute: () => T): T { + return gates.run(OUTSIDE_MODEL_GATE, execute) +} + +/** + * Whether this call must classify what it returns: Chat's own principals, under the gate of the + * surface they read through. Chat with no gate is refused here, before anything loads, rather + * than handed content no surface classified. Use cases read provenance only when this is true, so + * a caller no gate observes — the UI, API keys, MCP, the executor, and app tools Chat runs under an + * executor identity, whose results carry their own provenance metadata — issues no extra query + * and sees no change. + */ +export function mustClassifyContent(principal: Principal): boolean { + if (!isCopilotPrincipal(principal)) return false + if (!gates.getStore()) throw new ContentDeliveryUnobservedError() + return true +} + +/** The per-item verdicts for one call to {@link admitContent}. */ +export class ContentAdmission { + private readonly verdicts: ReadonlyMap + + constructor(sources: readonly ContentSource[], verdicts: readonly boolean[]) { + this.verdicts = new Map(sources.map((source, index) => [source, verdicts[index] === true])) + } + + admits(source: ContentSource): boolean { + return this.verdicts.get(source) === true + } + + get refused(): ContentSource[] { + return [...this.verdicts].filter(([, admitted]) => !admitted).map(([source]) => source) + } +} + +/** + * Accounts for the operation running now and the requests around it. An enclosing operation is + * not marked: a nested use case's admission covers what it returned, not what the outer one adds. + */ +function markAccounted(): void { + const current = ledgers.getStore() + if (!current) return + current.accounted = true + for (let ledger = current.parent; ledger; ledger = ledger.parent) { + if (ledger.declaration === undefined) ledger.accounted = true + } +} + +function mergeContentProvenance( + values: Iterable +): ContentProvenance | undefined { + let admitted = false + let unrecorded = false + const merged = mergeDurableSecretProvenanceSources( + (function* () { + for (const value of values) { + admitted = true + if (value.status === 'unrecorded') unrecorded = true + else yield value + } + })() + ) + if (!admitted) return undefined + if (unrecorded && merged.status === 'exact' && merged.entries.length === 0) { + return { status: 'unrecorded' } + } + return merged +} + +/** A committed write returned an item of `kind` its gate refused or could not decide. */ +function withholdAfterWrite(kind: ContentSourceKind): void { + for (let ledger = ledgers.getStore(); ledger; ledger = ledger.parent) { + if (ledger.withheldAfterWrite === undefined || ledger.withheldAfterWrite === 'tool-result') { + ledger.withheldAfterWrite = kind + } + } +} + +/** + * Classifies every item a use case is about to return, before its content loads. Call it once per + * result, with every contributing item, including an empty list: that call is the use case's + * promise to the operation's `sourced` declaration that nothing else in the result is sourced. + * + * Chat with no gate installed is refused (fail closed); every other caller is admitted unread. An + * empty list loads nothing, so it only accounts. + */ +export async function admitContent( + principal: Principal, + sources: readonly ContentSource[] +): Promise { + if ( + sources.length > 0 && + isCopilotPrincipal(principal) && + ledgers.getStore()?.declaration === 'none' + ) { + throw new ContentDeliveryUndeclaredError() + } + /** Accounting follows a decision: a caller that catches a failure has classified nothing. */ + if (sources.length === 0 || !mustClassifyContent(principal)) { + markAccounted() + return new ContentAdmission( + sources, + sources.map(() => true) + ) + } + const gate = gates.getStore() + if (!gate) throw new ContentDeliveryUnobservedError() + const ledger = ledgers.getStore() + let verdicts: readonly boolean[] + try { + verdicts = await gate.admit(sources) + } catch (error) { + /** Kept for the transport: a route maps the error to its own response and hides the cause. */ + for (let current: ContentLedger | undefined = ledger; current; current = current.parent) { + current.gateFailure ??= error + } + throw error + } + markAccounted() + if (ledger) { + const batch = mergeContentProvenance( + (function* () { + for (let index = 0; index < sources.length; index++) { + if (verdicts[index]) yield sources[index].provenance + } + })() + ) + if (batch) { + for (let current: ContentLedger | undefined = ledger; current; current = current.parent) { + const previous = current.admitted[0] + const aggregate = previous ? mergeContentProvenance([previous, batch]) : batch + if (aggregate) current.admitted.splice(0, current.admitted.length, aggregate) + } + } + } + return new ContentAdmission(sources, verdicts) +} + +/** + * Admits what a write returns after it committed. A refused item does not end the call here, so + * the operation still records its audit and signals its change; it then answers with a refusal + * naming the saved change instead of a result carrying content its gate refused. + */ +export async function admitWrittenContent( + principal: Principal, + sources: readonly ContentSource[] +): Promise { + let admission: ContentAdmission + try { + admission = await admitContent(principal, sources) + } catch (error) { + /** + * A gate that cannot decide (its record could not be written) must not undo the write's audit + * and effects either: every item is withheld, and the operation answers "saved, not shown". + * A transport that must answer the failure itself reads it from the request's ledger. + */ + const [first] = sources + if (!ledgers.getStore() || !first) throw error + /** The result is decided after all: withheld whole, as saved-not-shown. */ + markAccounted() + withholdAfterWrite(first.kind) + return new ContentAdmission( + sources, + sources.map(() => false) + ) + } + const refused = admission.refused[0] + if (refused) withholdAfterWrite(refused.kind) + return admission +} + +/** + * Admits each of many items on its own and returns the admitted ones, in order. `classify` maps + * the items to their sources, one per item and batched; it runs only when this call must + * classify, so any other caller pays no provenance read. An item it cannot vouch for is `unknown`. + */ +export async function admitEach( + principal: Principal, + items: readonly T[], + classify: (items: readonly T[]) => Promise, + options: { written?: ContentSourceKind } = {} +): Promise { + if (items.length === 0 || !mustClassifyContent(principal)) { + await admitContent(principal, []) + return [...items] + } + if (!options.written) { + const sources = await classify(items) + const admission = await admitContent(principal, sources) + return items.filter((_, index) => admission.admits(sources[index])) + } + /** A committed write whose items cannot even be classified is saved, then withheld whole. */ + let sources: readonly ContentSource[] + try { + sources = await classify(items) + } catch (error) { + if (!ledgers.getStore()) throw error + markAccounted() + withholdAfterWrite(options.written) + return [] + } + const admission = await admitWrittenContent(principal, sources) + return items.filter((_, index) => admission.admits(sources[index])) +} + +/** + * The single-item form: a refused item ends the call with a `403` before its content loads, with a + * constant message that names no content. + */ +export async function requireAdmittedContent( + principal: Principal, + source: ContentSource +): Promise { + const admission = await admitContent(principal, [source]) + if (!admission.admits(source)) throw new ContentRefusedError(source.kind) +} + +/** The gate failed during a dispatched request, which may have committed a write first. */ +export class ContentGateFailure extends Error { + constructor( + readonly gateFailure: unknown, + readonly committedWrite: boolean + ) { + super('The content gate failed during the request', { cause: gateFailure }) + this.name = 'ContentGateFailure' + } +} + +/** What one dispatched request delivered, as the transport that dispatched it needs to know. */ +export interface ContentDeliveryRecord { + result: T + /** Chat ran a `sourced` operation whose use case never accounted for its content. */ + unaccounted: boolean + /** The gate failed to decide during the request; see {@link ContentLedger.gateFailure}. */ + gateFailure?: unknown + /** A write committed, then its returned content was withheld: the change happened. */ + committedWrite: boolean + /** Bounded aggregate of admitted provenance, updated in place during deferred delivery. */ + admitted: readonly ContentProvenance[] +} + +/** + * Dispatches one request under its own ledger. A transport refuses a successful response whose + * record is `unaccounted`, whatever the HTTP method, so a route that forgot to classify its content + * fails closed instead of passing it through. + */ +export async function trackContentDelivery( + execute: () => Promise +): Promise> { + const ledger: ContentLedger = { + parent: ledgers.getStore(), + sourcedOperationAdmitted: false, + accounted: false, + admitted: [], + } + let result: T + try { + result = await ledgers.run(ledger, execute) + } catch (error) { + /** The request threw because its gate failed; its transport answers that, not the error. */ + if (ledger.gateFailure === undefined) throw error + throw new ContentGateFailure( + ledger.gateFailure, + ledger.withheldAfterWrite !== undefined && ledger.withheldAfterWrite !== 'tool-result' + ) + } + return { + result, + unaccounted: ledger.sourcedOperationAdmitted && !ledger.accounted, + committedWrite: + ledger.withheldAfterWrite !== undefined && ledger.withheldAfterWrite !== 'tool-result', + admitted: ledger.admitted, + ...(ledger.gateFailure === undefined ? {} : { gateFailure: ledger.gateFailure }), + } +} + +/** Route admission records that Chat was admitted to an operation, in the request's ledger. */ +export function noteContentDeliveryOperation( + operation: ApplicationOperation, + principal: Principal +): void { + if (!isCopilotPrincipal(principal) || operation.contentDelivery !== 'sourced') return + for (let ledger = ledgers.getStore(); ledger; ledger = ledger.parent) { + ledger.sourcedOperationAdmitted = true + } +} + +/** + * Runs one use case for Chat under its operation's declaration: `execute` is the business phase, + * `complete` its audit and after-success effects. A `sourced` operation that returns without + * accounting throws rather than hand its content to a model unclassified, and one whose committed + * write returned refused content answers with that refusal once its effects are complete. This + * holds on every Chat path, transported or called in-process by a tool. + */ +export async function runDeclaredContentOperation( + operation: ApplicationOperation, + principal: Principal, + execute: () => Promise, + complete?: (result: T) => Promise +): Promise { + if (!isCopilotPrincipal(principal)) { + const result = await execute() + await complete?.(result) + return result + } + noteContentDeliveryOperation(operation, principal) + const ledger: ContentLedger = { + parent: ledgers.getStore(), + declaration: operation.contentDelivery, + sourcedOperationAdmitted: false, + accounted: false, + admitted: [], + } + const result = await ledgers.run(ledger, execute) + await complete?.(result) + if (operation.contentDelivery === 'sourced' && !ledger.accounted) { + throw new ContentDeliveryUnaccountedError(operation.id) + } + if (ledger.withheldAfterWrite) { + throw new ContentRefusedError(ledger.withheldAfterWrite, { afterWrite: true }) + } + return result +} diff --git a/apps/sim/lib/core/application/index.ts b/apps/sim/lib/core/application/index.ts index 48215ea637a..5b91b1fd8f2 100644 --- a/apps/sim/lib/core/application/index.ts +++ b/apps/sim/lib/core/application/index.ts @@ -61,3 +61,26 @@ export { type WorkspaceOperation, } from '@/lib/core/application/workspace-operation' export { PermissionGroupCapabilityError } from '@/lib/permission-groups/capability-error' +export { + admitContent, + admitEach, + admitWrittenContent, + type ContentDeliveryDeclaration, + type ContentDeliveryGate, + ContentDeliveryUnobservedError, + ContentGateFailure, + type ContentProvenance, + ContentRefusedError, + type ContentSource, + type ContentSourceKind, + type ContentView, + isCopilotPrincipal, + mustClassifyContent, + noteContentDeliveryOperation, + observeContentDelivery, + observeContentOutsideModel, + redactBeforeCutting, + requireAdmittedContent, + runDeclaredContentOperation, + trackContentDelivery, +} from './content-delivery' diff --git a/apps/sim/lib/core/application/operation.ts b/apps/sim/lib/core/application/operation.ts index ba2721ca1ae..a35b3522159 100644 --- a/apps/sim/lib/core/application/operation.ts +++ b/apps/sim/lib/core/application/operation.ts @@ -1,5 +1,6 @@ import type { Principal } from '@sim/auth/principal' import type { OAuthApiScope } from '@/lib/auth/oauth-provider' +import type { ContentDeliveryDeclaration } from '@/lib/core/application/content-delivery' import { requireOAuthOperationScope } from '@/lib/core/application/oauth-authorization' import type { OrchestrationRequestContext } from '@/lib/core/orchestration/types' import { @@ -68,6 +69,12 @@ export interface ApplicationOperation { * and `check:permission-group-enforcement` keeps reading the source. */ readonly capability: OperationDeclarableCapability | 'none' + /** + * Whether the result can carry content derived from a provenance-bearing store. Required on + * every workspace operation Chat can reach (see {@link ContentDeliveryDeclaration}); a `sourced` + * operation's use case classifies each item through `admitContent` before returning it to Chat. + */ + readonly contentDelivery?: ContentDeliveryDeclaration } export type OAuthOperationPolicy = @@ -242,3 +249,27 @@ export interface OperationUseCase { request?: OrchestrationRequestContext }): Promise } + +/** + * An operation Chat can reach says whether it returns stored content; one it cannot reach says + * nothing, so the field always means a reviewed answer. Test fixtures escape type-checking, so + * this is enforced where operations are minted too. + */ +export function assertContentDeliveryDeclaration( + operation: ApplicationOperation, + delegatedServices: readonly string[] +): void { + const admitsCopilot = delegatedServices.includes('copilot') + if ( + admitsCopilot && + operation.contentDelivery !== 'sourced' && + operation.contentDelivery !== 'none' + ) { + throw new Error( + `Operation ${operation.id} admits Copilot but does not declare contentDelivery; name 'sourced' or 'none'` + ) + } + if (!admitsCopilot && operation.contentDelivery !== undefined) { + throw new Error(`Operation ${operation.id} declares contentDelivery without admitting Copilot`) + } +} diff --git a/apps/sim/lib/core/application/organization-authorization.test.ts b/apps/sim/lib/core/application/organization-authorization.test.ts index 019c963a649..4a727ea8763 100644 --- a/apps/sim/lib/core/application/organization-authorization.test.ts +++ b/apps/sim/lib/core/application/organization-authorization.test.ts @@ -21,6 +21,7 @@ const mocks = { const principal = createSessionPrincipal({ userId: 'member', sessionId: 'session' }) const operation = defineOrganizationOperation({ id: 'search.read', + contentDelivery: 'none', minimumRole: 'member', principalKinds: ['session', 'personal_api_key', 'oauth_access_token', 'organization_delegated'], oauthScope: 'api:read', diff --git a/apps/sim/lib/core/application/organization-operation.ts b/apps/sim/lib/core/application/organization-operation.ts index 8f2cb7695c6..b359f5fb192 100644 --- a/apps/sim/lib/core/application/organization-operation.ts +++ b/apps/sim/lib/core/application/organization-operation.ts @@ -1,6 +1,7 @@ import type { OrganizationDelegatedPrincipal, Principal } from '@sim/auth/principal' import type { ApplicationOperation } from '@/lib/core/application/operation' import { + assertContentDeliveryDeclaration, assertOperationCapability, assertOperationOAuthPolicy, } from '@/lib/core/application/operation' @@ -44,6 +45,7 @@ export function defineOrganizationOperation { it('preserves and freezes an explicit delegated service allowlist', () => { const operation = defineWorkspaceOperation({ id: 'test.read', + contentDelivery: 'none', minimumRole: 'read', workspaceApiKey: 'deny', principalKinds: ['delegated'], @@ -33,6 +34,7 @@ describe('defineWorkspaceOperation delegated service policy', () => { expect(() => defineWorkspaceOperation({ id: 'test.unused_service_policy', + contentDelivery: 'none', minimumRole: 'read', workspaceApiKey: 'deny', principalKinds: ['session'], @@ -46,6 +48,7 @@ describe('defineWorkspaceOperation delegated service policy', () => { expect(() => defineWorkspaceOperation({ id: 'test.duplicate_service_policy', + contentDelivery: 'none', minimumRole: 'read', workspaceApiKey: 'deny', principalKinds: ['delegated'], diff --git a/apps/sim/lib/core/application/workspace-operation.ts b/apps/sim/lib/core/application/workspace-operation.ts index 984bac44ddc..037c5d90025 100644 --- a/apps/sim/lib/core/application/workspace-operation.ts +++ b/apps/sim/lib/core/application/workspace-operation.ts @@ -1,11 +1,13 @@ import type { DelegatedPrincipal, DelegatedServiceId, Principal } from '@sim/auth/principal' import type { PermissionType } from '@sim/platform-authz/workspace' +import type { ContentDeliveryDeclaration } from '@/lib/core/application/content-delivery' import type { ApplicationOperation, OAuthOperationPolicy, PrincipalKind, } from '@/lib/core/application/operation' import { + assertContentDeliveryDeclaration, assertOperationCapability, assertOperationOAuthPolicy, } from '@/lib/core/application/operation' @@ -69,6 +71,19 @@ type DelegatedPrincipalConsistency< } : { readonly delegatedServices?: never } +/** + * Chat delivers whatever a Copilot-admitted operation returns to its model, so each one says + * whether that can be stored content; no other operation may claim to. + */ +type ContentDeliveryConsistency< + PrincipalKinds extends readonly PrincipalKind[], + DelegatedServices extends readonly DelegatedServiceId[], +> = 'delegated' extends PrincipalKinds[number] + ? 'copilot' extends DelegatedServices[number] + ? { readonly contentDelivery: ContentDeliveryDeclaration } + : { readonly contentDelivery?: never } + : { readonly contentDelivery?: never } + type ResourcePolicyOperationConsistency = Binding extends ResourcePolicyBinding ? { readonly resourcePolicy: Binding } @@ -85,6 +100,7 @@ export function defineWorkspaceOperation< OAuthOperationPolicy & WorkspaceApiKeyPrincipalConsistency & DelegatedPrincipalConsistency & + ContentDeliveryConsistency & ResourcePolicyOperationConsistency ): WorkspaceOperation & OAuthOperationPolicy & @@ -120,6 +136,8 @@ export function defineWorkspaceOperation< if (operation.resourcePolicy) requireResourcePolicyBinding(operation.resourcePolicy) + assertContentDeliveryDeclaration(operation, delegatedServices) + /** * `capability` is required, so refusing an absent one reads as unreachable. * It is not. `apps/sim/tsconfig.json` excludes test files from type-checking, diff --git a/apps/sim/lib/credential-groups/application/operations.ts b/apps/sim/lib/credential-groups/application/operations.ts index c3c6e40c01b..a6db72f9f9e 100644 --- a/apps/sim/lib/credential-groups/application/operations.ts +++ b/apps/sim/lib/credential-groups/application/operations.ts @@ -12,6 +12,7 @@ export const credentialGroupOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'none', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }), diff --git a/apps/sim/lib/credential-groups/application/organization-access.ts b/apps/sim/lib/credential-groups/application/organization-access.ts index f56e84e9952..f05d749ee24 100644 --- a/apps/sim/lib/credential-groups/application/organization-access.ts +++ b/apps/sim/lib/credential-groups/application/organization-access.ts @@ -27,6 +27,7 @@ import { export const organizationAccountAccessOperations = { read: defineOrganizationOperation({ id: 'organization_accounts.workspace_access.read', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -35,6 +36,7 @@ export const organizationAccountAccessOperations = { }), update: defineOrganizationOperation({ id: 'organization_accounts.workspace_access.update', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', diff --git a/apps/sim/lib/credential-groups/application/organization-account-management.ts b/apps/sim/lib/credential-groups/application/organization-account-management.ts index a091e03c448..cb3ea051659 100644 --- a/apps/sim/lib/credential-groups/application/organization-account-management.ts +++ b/apps/sim/lib/credential-groups/application/organization-account-management.ts @@ -25,6 +25,7 @@ import { evictMcpServerConnections } from '@/lib/mcp/connection-pool' export const organizationAccountManagementOperations = { people: defineOrganizationOperation({ id: 'organization_accounts.people.list', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -33,6 +34,7 @@ export const organizationAccountManagementOperations = { }), invite: defineOrganizationOperation({ id: 'organization_accounts.people.invite', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -41,6 +43,7 @@ export const organizationAccountManagementOperations = { }), resend: defineOrganizationOperation({ id: 'organization_accounts.people.resend', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -49,6 +52,7 @@ export const organizationAccountManagementOperations = { }), revoke: defineOrganizationOperation({ id: 'organization_accounts.people.revoke', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -57,6 +61,7 @@ export const organizationAccountManagementOperations = { }), addMcp: defineOrganizationOperation({ id: 'organization_accounts.mcp.add', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -65,6 +70,7 @@ export const organizationAccountManagementOperations = { }), removeMcp: defineOrganizationOperation({ id: 'organization_accounts.mcp.remove', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', diff --git a/apps/sim/lib/credential-groups/application/organization-accounts.ts b/apps/sim/lib/credential-groups/application/organization-accounts.ts index 3cfe7c5e6e0..5158f45ba45 100644 --- a/apps/sim/lib/credential-groups/application/organization-accounts.ts +++ b/apps/sim/lib/credential-groups/application/organization-accounts.ts @@ -52,6 +52,7 @@ import { isKnowledgeMemberAccessAvailable } from '@/lib/knowledge/access/availab export const organizationAccountOperations = { read: defineOrganizationOperation({ id: 'organization_accounts.read', + contentDelivery: 'none', minimumRole: 'member', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -60,6 +61,7 @@ export const organizationAccountOperations = { }), ensure: defineOrganizationOperation({ id: 'organization_accounts.ensure', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -68,6 +70,7 @@ export const organizationAccountOperations = { }), update: defineOrganizationOperation({ id: 'organization_accounts.update', + contentDelivery: 'none', minimumRole: 'admin', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', @@ -76,6 +79,7 @@ export const organizationAccountOperations = { }), connect: defineOrganizationOperation({ id: 'organization_accounts.connect', + contentDelivery: 'none', minimumRole: 'member', principalKinds: ['session', 'organization_delegated'], delegationAudience: 'sim:settings', diff --git a/apps/sim/lib/credentials/application/operations.ts b/apps/sim/lib/credentials/application/operations.ts index 566e5534e1e..b1bb906658b 100644 --- a/apps/sim/lib/credentials/application/operations.ts +++ b/apps/sim/lib/credentials/application/operations.ts @@ -44,6 +44,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), 'member' @@ -61,6 +62,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), authorizePersonal: defineWorkspaceOperation({ @@ -69,6 +71,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), listInternal: defineWorkspaceOperation({ @@ -84,6 +87,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'integrations.manage', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -99,6 +103,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'integrations.manage', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', @@ -125,6 +130,7 @@ export const credentialOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', principalKinds: ['session', 'personal_api_key', 'oauth_access_token', 'delegated'], delegatedServices: ['copilot'], }), @@ -133,6 +139,7 @@ export const credentialOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), @@ -142,6 +149,7 @@ export const credentialOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), read: defineCredentialOperation( @@ -168,6 +176,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), 'admin' @@ -179,6 +188,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), 'admin' @@ -196,6 +206,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), upsertMember: defineCredentialOperation( @@ -205,6 +216,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), 'admin' @@ -216,6 +228,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPALS, }), 'admin' @@ -239,6 +252,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'integrations.manage', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['executor', 'copilot'], resourcePolicy: { @@ -251,6 +265,7 @@ export const credentialOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'mcp_tools.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['executor', 'copilot'], resourcePolicy: { diff --git a/apps/sim/lib/credentials/application/resolve-organization-personal-token.ts b/apps/sim/lib/credentials/application/resolve-organization-personal-token.ts index 2fc9b72a254..124427d6104 100644 --- a/apps/sim/lib/credentials/application/resolve-organization-personal-token.ts +++ b/apps/sim/lib/credentials/application/resolve-organization-personal-token.ts @@ -23,6 +23,7 @@ import { livePolicyFor, loadLiveSearchPolicies } from '@/lib/sim-search/live/pol /** Search's organization delegation can use only the member's currently available personal grant. */ export const organizationPersonalCredentialOperation = defineOrganizationOperation({ id: 'credentials.organization.personal.use', + contentDelivery: 'none', minimumRole: 'member', principalKinds: ['organization_delegated'], delegatedServices: ['copilot'], @@ -132,6 +133,7 @@ export const resolveOrganizationPersonalToken = { export const organizationPersonalConnectionOperation = defineOrganizationOperation({ id: 'credentials.organization.personal.prepareConnection', + contentDelivery: 'none', minimumRole: 'member', principalKinds: ['organization_delegated'], delegatedServices: ['copilot'], diff --git a/apps/sim/lib/custom-tools/application/operations.ts b/apps/sim/lib/custom-tools/application/operations.ts index d00ab6124bc..c6a238f0694 100644 --- a/apps/sim/lib/custom-tools/application/operations.ts +++ b/apps/sim/lib/custom-tools/application/operations.ts @@ -28,6 +28,7 @@ export const customToolOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'custom_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), listAvailable: defineWorkspaceOperation({ @@ -36,6 +37,7 @@ export const customToolOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'custom_tools.use', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), read: defineWorkspaceOperation({ @@ -44,6 +46,7 @@ export const customToolOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'custom_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), readAvailableByIdOrTitle: defineWorkspaceOperation({ @@ -51,6 +54,7 @@ export const customToolOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'custom_tools.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot', 'executor'], }), @@ -60,6 +64,7 @@ export const customToolOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'custom_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), update: defineWorkspaceOperation({ @@ -68,6 +73,7 @@ export const customToolOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'custom_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), delete: defineWorkspaceOperation({ @@ -76,6 +82,7 @@ export const customToolOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'custom_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), } as const diff --git a/apps/sim/lib/dashboards/application/availability.ts b/apps/sim/lib/dashboards/application/availability.ts index 758dfeafbc9..d5ecc8008a3 100644 --- a/apps/sim/lib/dashboards/application/availability.ts +++ b/apps/sim/lib/dashboards/application/availability.ts @@ -9,6 +9,7 @@ const availabilityOperation = defineWorkspaceOperation({ oauthScope: 'api:read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [ 'session', 'personal_api_key', diff --git a/apps/sim/lib/dashboards/application/operations.ts b/apps/sim/lib/dashboards/application/operations.ts index c66a54ccd43..d9d697e3acd 100644 --- a/apps/sim/lib/dashboards/application/operations.ts +++ b/apps/sim/lib/dashboards/application/operations.ts @@ -11,12 +11,14 @@ export const dashboardOperations = { read: defineWorkspaceOperation({ ...policy, capability: 'files.use', + contentDelivery: 'none', id: 'dashboards.read', minimumRole: 'read', }), save: defineWorkspaceOperation({ ...policy, capability: 'files.use', + contentDelivery: 'none', id: 'dashboards.save', minimumRole: 'write', }), diff --git a/apps/sim/lib/data-drains/application/operations.ts b/apps/sim/lib/data-drains/application/operations.ts index dd21c6c3848..4c5c62c2f5f 100644 --- a/apps/sim/lib/data-drains/application/operations.ts +++ b/apps/sim/lib/data-drains/application/operations.ts @@ -13,48 +13,56 @@ export const dataDrainOperations = Object.freeze({ // permission-group-exempt: organization administrators govern organization-wide exports. list: defineOrganizationOperation({ id: 'organization.data_drains.list', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. get: defineOrganizationOperation({ id: 'organization.data_drains.get', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. create: defineOrganizationOperation({ id: 'organization.data_drains.create', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. update: defineOrganizationOperation({ id: 'organization.data_drains.update', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. delete: defineOrganizationOperation({ id: 'organization.data_drains.delete', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. test: defineOrganizationOperation({ id: 'organization.data_drains.test', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. run: defineOrganizationOperation({ id: 'organization.data_drains.run', + contentDelivery: 'none', ...policy, capability: 'none', }), // permission-group-exempt: organization administrators govern organization-wide exports. runs: defineOrganizationOperation({ id: 'organization.data_drains.runs', + contentDelivery: 'none', ...policy, capability: 'none', }), diff --git a/apps/sim/lib/execution/durable-secret-provenance-telemetry.ts b/apps/sim/lib/execution/durable-secret-provenance-telemetry.ts index 59d8ef72e4e..49c4c19d9f3 100644 --- a/apps/sim/lib/execution/durable-secret-provenance-telemetry.ts +++ b/apps/sim/lib/execution/durable-secret-provenance-telemetry.ts @@ -9,7 +9,12 @@ import { createLogger } from '@sim/logger' const persistenceLogger = createLogger('DurableSecretProvenancePersistence') -export type DurableSecretProvenanceSurface = 'memory' | 'table-row' | 'knowledge' | 'workspace-file' +export type DurableSecretProvenanceSurface = + | 'memory' + | 'table-row' + | 'knowledge' + | 'workspace-file' + | 'tool-result' export type DurableSecretProvenanceWriteCause = | 'source-provenance-unknown' @@ -35,6 +40,11 @@ export type DurableSecretProvenanceRefusalCause = | 'workspace-file-provenance-unavailable' | 'workspace-file-opaque-secret-content' | 'workspace-file-registry-unavailable' + | 'table-row-provenance-unavailable' + | 'table-row-opaque-secret-content' + | 'knowledge-opaque-secret-content' + | 'content-registry-unavailable' + | 'tool-result-provenance-unavailable' export interface DurableSecretProvenanceRefusalReport { surface: DurableSecretProvenanceSurface diff --git a/apps/sim/lib/execution/durable-secret-provenance.ts b/apps/sim/lib/execution/durable-secret-provenance.ts index 980c666298f..8aa351aaaa5 100644 --- a/apps/sim/lib/execution/durable-secret-provenance.ts +++ b/apps/sim/lib/execution/durable-secret-provenance.ts @@ -5,8 +5,10 @@ import { isPrivateSecretProvenanceBundleV1, type PrivateSecretProvenanceBundleV1, } from '@/lib/execution/model-input-provenance' -import { SecretProvenanceBudget } from '@/lib/execution/provenance-budget' -import { PROVENANCE_MAX_SERIALIZED_BYTES } from '@/lib/execution/provenance-limits' +import { + mergeDurableSecretProvenanceSources, + normalizeDurableSecretProvenanceBindings, +} from '@/lib/execution/provenance-entries' import { type ResolvedSecretTraceProvenanceV1, ResolvedSecretTraceRegistry, @@ -32,59 +34,6 @@ export function normalizeDurableSecretProvenanceEntries( return Array.isArray(value) ? normalizeDurableSecretProvenanceBindings(value) : undefined } -function normalizeDurableSecretProvenanceBindings( - candidates: Iterable -): DurableSecretProvenanceEntry[] | undefined { - const entries = new Map() - const budget = new SecretProvenanceBudget() - for (const candidate of candidates) { - if (!candidate || typeof candidate !== 'object' || Array.isArray(candidate)) return undefined - const record = candidate as Record - if ( - typeof record.encryptedValue !== 'string' || - record.encryptedValue.length === 0 || - (record.name !== undefined && typeof record.name !== 'string') || - (record.sourceUserId !== undefined && typeof record.sourceUserId !== 'string') || - (record.sourceWorkspaceId !== undefined && typeof record.sourceWorkspaceId !== 'string') || - (record.sourceValueHash !== undefined && typeof record.sourceValueHash !== 'string') || - (record.sourceWorkspaceId !== undefined && record.sourceUserId === undefined) - ) { - return undefined - } - const entry: DurableSecretProvenanceEntry = { - encryptedValue: record.encryptedValue, - ...(typeof record.name === 'string' && record.name.length > 0 ? { name: record.name } : {}), - ...(typeof record.sourceUserId === 'string' && record.sourceUserId.length > 0 - ? { sourceUserId: record.sourceUserId } - : {}), - ...(typeof record.sourceWorkspaceId === 'string' && record.sourceWorkspaceId.length > 0 - ? { sourceWorkspaceId: record.sourceWorkspaceId } - : {}), - ...(typeof record.sourceValueHash === 'string' && record.sourceValueHash.length > 0 - ? { sourceValueHash: record.sourceValueHash } - : {}), - } - let minimumBytes = 0 - for (const field of Object.values(entry)) { - minimumBytes += Buffer.byteLength(field, 'utf8') - if (minimumBytes > PROVENANCE_MAX_SERIALIZED_BYTES) return undefined - } - const key = JSON.stringify(entry) - if (entries.has(key)) continue - if (!budget.add(entry.encryptedValue, Buffer.byteLength(key, 'utf8'))) return undefined - entries.set(key, entry) - } - - return [...entries.values()].sort( - (left, right) => - compareStrings(left.sourceUserId ?? '', right.sourceUserId ?? '') || - compareStrings(left.sourceWorkspaceId ?? '', right.sourceWorkspaceId ?? '') || - compareStrings(left.sourceValueHash ?? '', right.sourceValueHash ?? '') || - compareStrings(left.name ?? '', right.name ?? '') || - compareStrings(left.encryptedValue, right.encryptedValue) - ) -} - /** Converts a complete transport envelope into its scope-preserving durable representation. */ export function durableSecretProvenanceFromEnvelope( provenance: ResolvedSecretTraceProvenanceV1 @@ -155,15 +104,7 @@ export function durableSecretProvenanceFromPrivateBundle( export function mergeDurableSecretProvenance( ...values: readonly DurableSecretProvenance[] ): DurableSecretProvenance { - if (values.some((value) => value.status === 'unknown')) return { status: 'unknown' } - const normalized = normalizeDurableSecretProvenanceBindings( - (function* () { - for (const value of values) { - if (value.status === 'exact') yield* value.entries - } - })() - ) - return normalized ? { status: 'exact', entries: normalized } : { status: 'unknown' } + return mergeDurableSecretProvenanceSources(values) } /** Binds exact entries to one persisted sub-value so later slicing cannot activate dropped data. */ @@ -201,10 +142,11 @@ export function filterDurableSecretProvenanceBySourceValues( export async function importDurableSecretProvenance( registry: ResolvedSecretTraceRegistry, provenance: DurableSecretProvenance, - value?: unknown + value?: unknown, + options: { atomic?: boolean } = {} ): Promise { if (provenance.status === 'unknown') { - registry.markIncomplete('durable-provenance-unknown') + if (!options.atomic) registry.markIncomplete('durable-provenance-unknown') return false } const entries = normalizeDurableSecretProvenanceEntries(provenance.entries) @@ -213,7 +155,7 @@ export async function importDurableSecretProvenance( * Malformed is not unrecorded: a sidecar that exists but cannot be parsed is a fault, and no * enforcement policy relaxes it. */ - registry.markIncomplete('durable-provenance-malformed') + if (!options.atomic) registry.markIncomplete('durable-provenance-malformed') return false } @@ -244,14 +186,25 @@ export async function importDurableSecretProvenance( group.entries.set(JSON.stringify(transportEntry), transportEntry) } - let complete = true - for (const group of grouped.values()) { - const envelope: ResolvedSecretTraceProvenanceV1 = { - version: 1, - complete: true, - entries: [...group.entries.values()], - ...(group.scope ? { scope: group.scope } : {}), + function* envelopes(): Generator { + for (const group of grouped.values()) { + yield { + version: 1, + complete: true, + entries: [...group.entries.values()], + ...(group.scope ? { scope: group.scope } : {}), + } } + } + if (options.atomic) { + return registry.tryImportProvenance(envelopes(), { + trusted: true, + origin: 'durableProvenance.admission', + value, + }) + } + let complete = true + for (const envelope of envelopes()) { const imported = value === undefined ? await registry.importProvenance(envelope, { diff --git a/apps/sim/lib/execution/isolated-vm-worker.cjs b/apps/sim/lib/execution/isolated-vm-worker.cjs index 4d2ff9ce5f8..c947ad4abe8 100644 --- a/apps/sim/lib/execution/isolated-vm-worker.cjs +++ b/apps/sim/lib/execution/isolated-vm-worker.cjs @@ -951,6 +951,11 @@ async function executeTask(request, executionId) { // Capture the bridge reference in a closure so hardening can unset the // global without breaking already-installed brokers. const __ref = globalThis.__brokerRef; + const __brokerErrors = new WeakMap(); + const __rememberBrokerError = __brokerErrors.set.bind(__brokerErrors); + Object.defineProperty(globalThis, '__readBrokerErrorId', { + value: __brokerErrors.get.bind(__brokerErrors), writable: false, configurable: false + }); globalThis.__brokers = globalThis.__brokers || {}; for (const name of ${brokerNamesJson}) { globalThis.__brokers[name] = async (args) => { @@ -963,7 +968,9 @@ async function executeTask(request, executionId) { let response; try { response = JSON.parse(responseJson); } catch { throw new Error('Invalid broker response'); } if (typeof response.error === 'string') { - throw new Error(response.error || 'Broker call failed'); + const error = new Error(response.error || 'Broker call failed'); + __rememberBrokerError(error, response.brokerErrorId); + throw error; } return response.resultJson === undefined || response.resultJson === null ? null @@ -1016,7 +1023,7 @@ async function executeTask(request, executionId) { userResult = { success: true } } else { const wrappedUserCode = ` - (async () => { + (async () => { const __readBrokerErrorId = globalThis.__readBrokerErrorId; try { await (async () => { ${code} @@ -1029,6 +1036,7 @@ async function executeTask(request, executionId) { message: error && error.message ? error.message : String(error), name: error && error.name ? error.name : 'Error', stack: error && error.stack ? error.stack : '', + brokerErrorId: __readBrokerErrorId(error), }, }); } @@ -1056,7 +1064,12 @@ async function executeTask(request, executionId) { return { result: null, stdout: stdoutChunks.join(''), - error: convertToCompatibleError(userResult.errorInfo, code), + error: { + ...convertToCompatibleError(userResult.errorInfo, code), + ...(Number.isSafeInteger(userResult.errorInfo.brokerErrorId) + ? { brokerErrorId: userResult.errorInfo.brokerErrorId } + : {}), + }, timings, } } @@ -1233,7 +1246,13 @@ process.on('message', async (msg) => { if (pending) { clearTimeout(pending.timeout) pendingBrokerCalls.delete(msg.brokerId) - pending.resolve(JSON.stringify({ error: msg.error, resultJson: msg.resultJson })) + pending.resolve( + JSON.stringify({ + error: msg.error, + resultJson: msg.resultJson, + ...(typeof msg.error === 'string' ? { brokerErrorId: msg.brokerId } : {}), + }) + ) } } } catch (err) { diff --git a/apps/sim/lib/execution/isolated-vm.ts b/apps/sim/lib/execution/isolated-vm.ts index 8292b05beba..936186f9455 100644 --- a/apps/sim/lib/execution/isolated-vm.ts +++ b/apps/sim/lib/execution/isolated-vm.ts @@ -66,7 +66,7 @@ interface IsolatedVMTaskRequest { * Host-side broker handler invoked when isolate code calls a broker. * Registered per-request via `executeInIsolatedVM(..., { brokers })`. */ -export type IsolatedVMBrokerHandler = (args: unknown) => Promise +export type IsolatedVMBrokerHandler = (args: unknown, brokerId: number) => Promise export interface IsolatedVMExecutionOptions { /** Broker name → handler. Must cover every broker listed in `request.task.brokers`. */ @@ -104,6 +104,8 @@ interface IsolatedVMTaskTimings { } interface IsolatedVMError { + /** Existing broker call ID, set only when a task propagates that broker's error object. */ + brokerErrorId?: number message: string name: string stack?: string @@ -735,7 +737,7 @@ function handleBrokerMessage( } Promise.resolve() - .then(() => pending.runInOutboundScope(() => handler(args))) + .then(() => pending.runInOutboundScope(() => handler(args, brokerId))) .then((resultValue) => { if (pending.cancelled) { sendResponse({ error: 'Execution cancelled' }) diff --git a/apps/sim/lib/execution/provenance-entries.ts b/apps/sim/lib/execution/provenance-entries.ts new file mode 100644 index 00000000000..b7ed0bdef1c --- /dev/null +++ b/apps/sim/lib/execution/provenance-entries.ts @@ -0,0 +1,78 @@ +import type { DurableSecretProvenanceEntry } from '@sim/db/schema' +import { compareStrings } from '@sim/utils/string' +import type { DurableSecretProvenance } from '@/lib/execution/durable-secret-provenance' +import { SecretProvenanceBudget } from '@/lib/execution/provenance-budget' +import { PROVENANCE_MAX_SERIALIZED_BYTES } from '@/lib/execution/provenance-limits' + +/** Normalizes and folds streamed durable bindings within the existing provenance budget. */ +export function normalizeDurableSecretProvenanceBindings( + candidates: Iterable +): DurableSecretProvenanceEntry[] | undefined { + const entries = new Map() + const budget = new SecretProvenanceBudget() + for (const candidate of candidates) { + if (!candidate || typeof candidate !== 'object' || Array.isArray(candidate)) return undefined + const record = candidate as Record + if ( + typeof record.encryptedValue !== 'string' || + record.encryptedValue.length === 0 || + (record.name !== undefined && typeof record.name !== 'string') || + (record.sourceUserId !== undefined && typeof record.sourceUserId !== 'string') || + (record.sourceWorkspaceId !== undefined && typeof record.sourceWorkspaceId !== 'string') || + (record.sourceValueHash !== undefined && typeof record.sourceValueHash !== 'string') || + (record.sourceWorkspaceId !== undefined && record.sourceUserId === undefined) + ) { + return undefined + } + const entry: DurableSecretProvenanceEntry = { + encryptedValue: record.encryptedValue, + ...(typeof record.name === 'string' && record.name.length > 0 ? { name: record.name } : {}), + ...(typeof record.sourceUserId === 'string' && record.sourceUserId.length > 0 + ? { sourceUserId: record.sourceUserId } + : {}), + ...(typeof record.sourceWorkspaceId === 'string' && record.sourceWorkspaceId.length > 0 + ? { sourceWorkspaceId: record.sourceWorkspaceId } + : {}), + ...(typeof record.sourceValueHash === 'string' && record.sourceValueHash.length > 0 + ? { sourceValueHash: record.sourceValueHash } + : {}), + } + let minimumBytes = 0 + for (const field of Object.values(entry)) { + minimumBytes += Buffer.byteLength(field, 'utf8') + if (minimumBytes > PROVENANCE_MAX_SERIALIZED_BYTES) return undefined + } + const key = JSON.stringify(entry) + if (entries.has(key)) continue + if (!budget.add(entry.encryptedValue, Buffer.byteLength(key, 'utf8'))) return undefined + entries.set(key, entry) + } + + return [...entries.values()].sort( + (left, right) => + compareStrings(left.sourceUserId ?? '', right.sourceUserId ?? '') || + compareStrings(left.sourceWorkspaceId ?? '', right.sourceWorkspaceId ?? '') || + compareStrings(left.sourceValueHash ?? '', right.sourceValueHash ?? '') || + compareStrings(left.name ?? '', right.name ?? '') || + compareStrings(left.encryptedValue, right.encryptedValue) + ) +} + +/** Merges streamed source evidence within the shared budget without materializing the sources. */ +export function mergeDurableSecretProvenanceSources( + values: Iterable +): DurableSecretProvenance { + let unknown = false + const normalized = normalizeDurableSecretProvenanceBindings( + (function* () { + for (const value of values) { + if (value.status === 'unknown') { + unknown = true + return + } + yield* value.entries + } + })() + ) + return !unknown && normalized ? { status: 'exact', entries: normalized } : { status: 'unknown' } +} diff --git a/apps/sim/lib/execution/sandbox/__integration__/content-reads.integration.ts b/apps/sim/lib/execution/sandbox/__integration__/content-reads.integration.ts new file mode 100644 index 00000000000..c6cabb09ca8 --- /dev/null +++ b/apps/sim/lib/execution/sandbox/__integration__/content-reads.integration.ts @@ -0,0 +1,337 @@ +import { execFile } from 'node:child_process' +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { promisify } from 'node:util' +import { db } from '@sim/db' +import { auditLog, organization, user, workspace, workspaceFiles } from '@sim/db/schema' +import { deleteWorkspaceFixture } from '@sim/db/testing/workspace-fixtures' +import { toolsMock, toolsMockFns } from '@sim/testing/mocks/tools.mock' +import { generateId } from '@sim/utils/id' +import { eq, inArray } from 'drizzle-orm' +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' + +const storage = vi.hoisted(() => ({ root: '' })) +vi.mock('@/lib/uploads/core/setup.server', () => ({ + get UPLOAD_DIR_SERVER() { + return storage.root + }, +})) +vi.mock('@/tools', () => toolsMock) + +import { observeContentDelivery } from '@/lib/core/application' +import { encryptSecret } from '@/lib/core/security/encryption' +import { + createKnowledgeAclFixtureIds, + seedKnowledgeAclFixture, +} from '@/lib/knowledge/__integration__/seed-source-access-fixture' +import { createCopilotChatFilePrincipal } from '@/lib/mothership/auth/file-delegation' +import { createRegistryContentGate } from '@/lib/mothership/content-delivery' +import { executeFunctionExecute } from '@/lib/mothership/tools/handlers/function-execute' +import { compileDoc } from '@/lib/mothership/tools/server/files/doc-compile' +import { compileDocForWrite } from '@/lib/mothership/tools/server/files/workspace-file' +import { + updateWorkspaceFileContent, + uploadWorkspaceFile, +} from '@/lib/uploads/contexts/workspace/workspace-file-manager' +import { getBoundWorkspaceFileSecretProvenanceByIdentity } from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance' +import { projectResolvedSecretModelContent } from '@/executor/utils/resolved-secret-content-projection' +import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry' + +const execute = promisify(execFile) +const secret = 'synthetic-broker-value-987654321' + +describe('Chat admits the file revision used by a sandbox', () => { + const ids = createKnowledgeAclFixtureIds() + const chatId = generateId() + let fileId: string + let workbench: string + + beforeAll(async () => { + storage.root = await mkdtemp(join(tmpdir(), 'sim-broker-content-')) + workbench = await mkdtemp(join(tmpdir(), 'sim-input-content-')) + await seedKnowledgeAclFixture(ids) + const file = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from('public bytes'), + 'source.txt', + 'text/plain', + { notifyWorkspaceChange: false } + ) + fileId = file.id + }) + afterAll(async () => { + await db.delete(auditLog).where(eq(auditLog.workspaceId, ids.workspaceId)) + await deleteWorkspaceFixture(db, eq(workspace.id, ids.workspaceId)) + await db.delete(organization).where(eq(organization.id, ids.organizationId)) + await db.delete(user).where(inArray(user.id, [ids.aliceId, ids.bobId])) + await rm(storage.root, { recursive: true, force: true }) + await rm(workbench, { recursive: true, force: true }) + }) + + function compile(source: string, registry: ResolvedSecretTraceRegistry) { + return observeContentDelivery( + createRegistryContentGate({ + registry, + workspaceId: ids.workspaceId, + actorUserId: ids.aliceId, + }), + () => + compileDoc({ + source, + fileName: 'fixture.pdf', + workspaceId: ids.workspaceId, + filePrincipal: createCopilotChatFilePrincipal({ + userId: ids.aliceId, + workspaceId: ids.workspaceId, + chatId, + }), + }) + ) + } + function registry() { + return new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }) + } + + it('reads variable file references through the real isolated worker and invalidates cached source revisions', async () => { + const source = `const id = ${JSON.stringify(fileId)}; const data = await getFileBase64(id); pdf.addPage().drawText(data);` + const first = await compile(source, registry()) + expect(first.buffer.subarray(0, 4).toString()).toBe('%PDF') + expect(first.contributingFiles).toHaveLength(1) + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from('updated public bytes'), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { mode: 'replace', provenance: { status: 'exact', entries: [] } }, + } + ) + const second = await compile(source, registry()) + expect(second.contributingFiles?.[0].key).not.toBe(first.contributingFiles?.[0].key) + const revisions = [...(first.contributingFiles ?? []), ...(second.contributingFiles ?? [])] + for (const identities of [revisions, [...revisions].reverse()]) { + const bound = await getBoundWorkspaceFileSecretProvenanceByIdentity( + ids.workspaceId, + identities + ) + expect(bound.get(fileId)).toEqual({ status: 'unknown' }) + } + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from('unverified bytes'), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { mode: 'replace', provenance: { status: 'unknown' } }, + } + ) + await expect(compile(source, registry())).rejects.toThrow(/provenance|read safely/i) + }, 60_000) + + it('imports evidence before a broker response can appear in a compiler error', async () => { + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from(secret), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { + mode: 'replace', + provenance: { + status: 'exact', + entries: [ + { + name: 'BROKER_TOKEN', + sourceUserId: ids.aliceId, + sourceWorkspaceId: ids.workspaceId, + encryptedValue: (await encryptSecret(secret)).encrypted, + }, + ], + }, + }, + } + ) + const trace = registry() + let message = '' + try { + await compile( + `const id = ${JSON.stringify(fileId)}; const uri = await getFileBase64(id); throw new Error(Buffer.from(uri.split(',')[1], 'base64').toString());`, + trace + ) + } catch (error) { + if (!(error instanceof Error)) throw error + message = error.message + } + expect(message).toContain(secret) + const projected = projectResolvedSecretModelContent(message, trace) + if (!projected.safe) throw new Error('Expected a safely projected compiler error') + expect(projected.value).not.toContain(secret) + expect(projected.value).toContain('{{BROKER_TOKEN}}') + }, 60_000) + + it('admits broker inputs during legacy document-write validation before returning compiler errors', async () => { + const principal = createCopilotChatFilePrincipal({ + userId: ids.aliceId, + workspaceId: ids.workspaceId, + chatId, + }) + const validate = (source: string, trace: ResolvedSecretTraceRegistry) => + observeContentDelivery( + createRegistryContentGate({ + registry: trace, + workspaceId: ids.workspaceId, + actorUserId: ids.aliceId, + }), + () => + compileDocForWrite({ + source, + fileName: 'write-validation.pdf', + workspaceId: ids.workspaceId, + principal, + ownerKey: `user:${ids.aliceId}`, + fallbackMime: 'application/pdf', + }) + ) + + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from('unverified input'), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { mode: 'replace', provenance: { status: 'unknown' } }, + } + ) + const refused = await validate( + `const id = ${JSON.stringify(fileId)}; await getFileBase64(id); pdf.addPage();`, + registry() + ) + expect.soft(refused).toMatchObject({ ok: false }) + if (!refused.ok) expect(refused.message).toMatch(/provenance|read safely/i) + + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from(secret), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { + mode: 'replace', + provenance: { + status: 'exact', + entries: [ + { + name: 'BROKER_TOKEN', + sourceUserId: ids.aliceId, + sourceWorkspaceId: ids.workspaceId, + encryptedValue: (await encryptSecret(secret)).encrypted, + }, + ], + }, + }, + } + ) + const trace = registry() + const failed = await validate( + `const id = ${JSON.stringify(fileId)}; const uri = await getFileBase64(id); throw new Error(Buffer.from(uri.split(',')[1], 'base64').toString());`, + trace + ) + expect(failed.ok).toBe(false) + if (failed.ok) throw new Error('Expected the document source to fail') + expect(failed.message).toContain(secret) + const projected = projectResolvedSecretModelContent(failed.message, trace) + if (!projected.safe) throw new Error('Expected a safely projected compiler error') + expect(projected.value).not.toContain(secret) + expect(projected.value).toContain('{{BROKER_TOKEN}}') + }, 60_000) + + it('keeps successful legacy dynamic-input documents readable after their first compile', async () => { + const references = Array.from({ length: 501 }, (_, index) => ({ + id: `wf_${generateId()}`, + key: `workspace/${ids.workspaceId}/cache-reference-${generateId()}.txt`, + originalName: `reference-${index}.txt`, + userId: ids.aliceId, + workspaceId: ids.workspaceId, + context: 'workspace', + contentType: 'text/plain', + sizeBytes: 6, + })) + await db.insert(workspaceFiles).values(references) + await mkdir(join(storage.root, 'workspace', ids.workspaceId), { recursive: true }) + for (const reference of references) { + await writeFile(join(storage.root, reference.key), 'public') + } + const source = `const ids = ${JSON.stringify(references.map((file) => file.id))}; for (const id of ids) await getFileBase64(id); pdf.addPage().drawText('compiled all references');` + const first = await compile(source, registry()) + expect(first.buffer.subarray(0, 4).toString()).toBe('%PDF') + expect(first.contributingFiles).toHaveLength(references.length) + const second = await compile(source, registry()) + expect(second.buffer.subarray(0, 4).toString()).toBe('%PDF') + expect(second.contributingFiles).toHaveLength(references.length) + }, 120_000) + + it('does not run code with a stale local file after refusing its requested replacement', async () => { + await updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from('unverified replacement'), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { mode: 'replace', provenance: { status: 'unknown' } }, + } + ) + await writeFile(join(workbench, 'source.txt'), 'previous mounted version') + await writeFile(join(workbench, 'result.txt'), 'not executed') + toolsMockFns.mockExecuteTool.mockImplementation(async (_id, params: { code: string }) => { + const output = await execute(process.execPath, ['-e', params.code], { cwd: workbench }) + return { success: true, output } + }) + const context = { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + chatId, + toolCallId: generateId(), + workflowId: '', + copilotToolExecution: true, + resolvedSecretTraceRegistry: registry(), + } + const code = + "const fs = require('node:fs'); fs.writeFileSync('result.txt', fs.readFileSync('source.txt'));" + const result = await executeFunctionExecute( + { + code, + language: 'javascript', + inputs: { files: [{ path: fileId, sandboxPath: '/tmp/source.txt' }] }, + }, + context + ) + expect(await readFile(join(workbench, 'result.txt'), 'utf8')).toBe('not executed') + expect(result).toMatchObject({ + success: false, + output: { codeExecuted: false, excludedInputs: [{ path: '/tmp/source.txt' }] }, + }) + expect(context.resolvedSecretTraceRegistry.isComplete()).toBe(true) + const next = await executeFunctionExecute( + { code: "process.stdout.write('clock usable')", language: 'javascript' }, + context + ) + expect(next).toMatchObject({ success: true, output: { stdout: 'clock usable' } }) + }) +}) diff --git a/apps/sim/lib/execution/sandbox/brokers/workspace-file.ts b/apps/sim/lib/execution/sandbox/brokers/workspace-file.ts index 32277c4419b..6be2e8196df 100644 --- a/apps/sim/lib/execution/sandbox/brokers/workspace-file.ts +++ b/apps/sim/lib/execution/sandbox/brokers/workspace-file.ts @@ -8,6 +8,7 @@ import { fetchWorkspaceFileBuffer, getWorkspaceFile, } from '@/lib/uploads/contexts/workspace/workspace-file-manager' +import { readWorkspaceFileContent } from '@/lib/workspace-files/application/read-workspace-file-content' const logger = createLogger('SandboxWorkspaceFileBroker') @@ -34,8 +35,24 @@ export const workspaceFileBroker: SandboxBroker maxBytes) + throw new Error('Workspace file exceeds the sandbox image byte limit') ctx.onWorkspaceFileAccess?.({ fileId: record.id, key: record.key, diff --git a/apps/sim/lib/execution/sandbox/run-task.ts b/apps/sim/lib/execution/sandbox/run-task.ts index d28d5175d54..7e09020b196 100644 --- a/apps/sim/lib/execution/sandbox/run-task.ts +++ b/apps/sim/lib/execution/sandbox/run-task.ts @@ -1,5 +1,7 @@ +import { AsyncLocalStorage } from 'node:async_hooks' import { createLogger } from '@sim/logger' import { generateShortId } from '@sim/utils/id' +import { ContentRefusedError } from '@/lib/core/application' import { executeInIsolatedVM, type IsolatedVMBrokerHandler, @@ -20,6 +22,7 @@ export interface RunSandboxTaskOptions { signal?: AbortSignal /** Records canonical workspace files whose bytes were successfully read by a task broker. */ onWorkspaceFileAccess?: SandboxBrokerContext['onWorkspaceFileAccess'] + filePrincipal?: SandboxBrokerContext['filePrincipal'] /** Run id that brokers see as `ctx.requestId`, for callers that keep per-run broker state. */ requestId?: string } @@ -59,10 +62,23 @@ export async function runSandboxTask( workspaceId: input.workspaceId, requestId, onWorkspaceFileAccess: options.onWorkspaceFileAccess, + filePrincipal: options.filePrincipal, } const brokers: Record = {} + const runInCallerContext = AsyncLocalStorage.snapshot() + let brokerRefusals: Map | undefined for (const broker of task.brokers) { - brokers[broker.name] = (args) => broker.handle(brokerContext, args) + brokers[broker.name] = async (args, brokerId) => { + try { + return await runInCallerContext(() => broker.handle(brokerContext, args)) + } catch (error) { + if (error instanceof ContentRefusedError) { + brokerRefusals ??= new Map() + brokerRefusals.set(brokerId, error) + } + throw error + } + } } const request: IsolatedVMExecutionRequest = { @@ -120,6 +136,11 @@ export async function runSandboxTask( if (result.error.stack) err.stack = result.error.stack throw err } + const brokerRefusal = + result.error.brokerErrorId === undefined + ? undefined + : brokerRefusals?.get(result.error.brokerErrorId) + if (brokerRefusal) throw brokerRefusal throw new SandboxUserCodeError( result.error.message, result.error.name || 'SandboxTaskError', diff --git a/apps/sim/lib/execution/sandbox/types.ts b/apps/sim/lib/execution/sandbox/types.ts index ee7082ac094..eaa3066da29 100644 --- a/apps/sim/lib/execution/sandbox/types.ts +++ b/apps/sim/lib/execution/sandbox/types.ts @@ -1,3 +1,5 @@ +import type { Principal } from '@sim/auth/principal' + /** * Types for the sandbox task system. * @@ -24,6 +26,8 @@ export interface SandboxBroker { export interface SandboxBrokerContext { workspaceId: string requestId: string + /** Host-side identity used to authorize and admit each actual broker read. Never sent to the isolate. */ + filePrincipal?: Principal onWorkspaceFileAccess?: (identity: { fileId: string key: string diff --git a/apps/sim/lib/file-parsers/json-parser.test.ts b/apps/sim/lib/file-parsers/json-parser.test.ts index 0a4d110e087..99b779d75f4 100644 --- a/apps/sim/lib/file-parsers/json-parser.test.ts +++ b/apps/sim/lib/file-parsers/json-parser.test.ts @@ -33,4 +33,29 @@ describe('JSON parser complexity limits', () => { code: 'invalid_format', }) }) + + /** + * Text extraction errors reach Chat as the read's error, and egress redaction recognizes only + * whole secrets, so a message quoting a clipped excerpt of the file could carry part of one. + */ + it.each([ + [ + 'JSON', + parseJSONBuffer, + `{"config": {"key": sk_live_FIXTURE_0123456789abcdefghij${'x'.repeat(200)}}}`, + ], + [ + 'JSONL', + parseJSONLBuffer, + `{"ok": true}\n{"key": "${'x'.repeat(80)}sk_live_FIXTURE_0123456789abcdefghij`, + ], + ] as const)( + 'names where invalid %s fails without quoting the file', + async (_kind, parse, text) => { + const error = await parse(Buffer.from(text)).catch((caught: unknown) => caught) + expect(error).toMatchObject({ code: 'invalid_format' }) + expect(String((error as Error).message)).not.toContain('sk_l') + expect(String((error as Error).message)).not.toContain('xxxx') + } + ) }) diff --git a/apps/sim/lib/file-parsers/json-parser.ts b/apps/sim/lib/file-parsers/json-parser.ts index 562e4cbcd43..b8e9c236fe2 100644 --- a/apps/sim/lib/file-parsers/json-parser.ts +++ b/apps/sim/lib/file-parsers/json-parser.ts @@ -213,6 +213,18 @@ export function stripJsonComments(text: string): string { return out.replace(/,(\s*[}\]])/g, '$1') } +/** V8's excerpt of the input: `Unexpected token 's', "…{"key": sk_l"... is not valid JSON`. */ +const QUOTED_INPUT_EXCERPT = /,\s*(?:\.\.\.)?".*"(?:\.\.\.)?\s+is not valid JSON$/s + +/** + * A parse failure's message without the clipped excerpt of the input V8 quotes in it. The message + * reaches Chat as the read's error, where redaction matches whole secrets only, so an excerpt cut + * through a secret would carry the part that survived the cut. + */ +function describeSyntaxError(error: unknown): string { + return getErrorMessage(error, 'Unknown error').replace(QUOTED_INPUT_EXCERPT, '') +} + function parseJsonContent(buffer: Uint8Array): FileParseResult { const decoded = decodeTextBuffer(buffer) try { @@ -226,7 +238,7 @@ function parseJsonContent(buffer: Uint8Array): FileParseResult { if (lenient) return lenient throw new FileParserError( 'invalid_format', - `Invalid JSON: ${getErrorMessage(error, 'Unknown error')}`, + `Invalid JSON: ${describeSyntaxError(error)}`, error ) } @@ -282,7 +294,7 @@ function parseJsonLinesContent(buffer: Uint8Array): FileParseResult { } catch (error) { throw new FileParserError( 'invalid_format', - `Invalid JSONL on line ${lineNumber}: ${line.slice(0, 100)}`, + `Invalid JSONL on line ${lineNumber}: ${describeSyntaxError(error)}`, error ) } diff --git a/apps/sim/lib/function-execution/application/execute-chat-function.ts b/apps/sim/lib/function-execution/application/execute-chat-function.ts index ed083afddc4..ec70b8bce40 100644 --- a/apps/sim/lib/function-execution/application/execute-chat-function.ts +++ b/apps/sim/lib/function-execution/application/execute-chat-function.ts @@ -21,11 +21,13 @@ export const executeChatFunction = defineAuthorizedChatUseCase({ minimumRole: 'read', workspaceApiKey: 'deny', capability: 'copilot.use', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['copilot'], }), organizationOperation: defineOrganizationOperation({ id: 'function-executions.execute_chat', + contentDelivery: 'none', minimumRole: 'member', capability: 'copilot.use', principalKinds: ['organization_delegated'], diff --git a/apps/sim/lib/function-execution/application/operations.ts b/apps/sim/lib/function-execution/application/operations.ts index bae1baceacd..e899d8913f2 100644 --- a/apps/sim/lib/function-execution/application/operations.ts +++ b/apps/sim/lib/function-execution/application/operations.ts @@ -7,6 +7,7 @@ export const functionExecutionOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'none', + contentDelivery: 'none', principalKinds: ['delegated'], delegatedServices: ['executor', 'copilot'], }), diff --git a/apps/sim/lib/invitations/application/list-workspace-invitations.ts b/apps/sim/lib/invitations/application/list-workspace-invitations.ts index ad2b66a4c61..ffd094bb85e 100644 --- a/apps/sim/lib/invitations/application/list-workspace-invitations.ts +++ b/apps/sim/lib/invitations/application/list-workspace-invitations.ts @@ -8,6 +8,7 @@ export const listWorkspaceInvitationsOperation = defineWorkspaceOperation({ id: 'workspace_invitations.list', minimumRole: 'admin', capability: 'none', + contentDelivery: 'none', workspaceApiKey: 'deny', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], diff --git a/apps/sim/lib/invitations/application/manage-invitation.ts b/apps/sim/lib/invitations/application/manage-invitation.ts index 20b37b67a8e..3c04ae48435 100644 --- a/apps/sim/lib/invitations/application/manage-invitation.ts +++ b/apps/sim/lib/invitations/application/manage-invitation.ts @@ -27,6 +27,7 @@ export const invitationManagementOperations = { /** permission-group-exempt: cancellation removes access and remains available when sends are withheld. */ cancel: defineOrganizationOperation({ id: 'organization_invitations.cancel', + contentDelivery: 'none', minimumRole: 'admin', capability: 'none', principalKinds: ['session', 'organization_delegated'], @@ -36,6 +37,7 @@ export const invitationManagementOperations = { /** permission-group-exempt: resend checks invitations.send against each canonical admission and workspace scope below. */ resend: defineOrganizationOperation({ id: 'organization_invitations.resend', + contentDelivery: 'none', minimumRole: 'admin', capability: 'none', principalKinds: ['session', 'organization_delegated'], @@ -256,6 +258,7 @@ export const workspaceInvitationManagementOperations = { id: 'workspace_invitations.cancel', minimumRole: 'admin', capability: 'none', + contentDelivery: 'none', workspaceApiKey: 'deny', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], @@ -264,6 +267,7 @@ export const workspaceInvitationManagementOperations = { id: 'workspace_invitations.resend', minimumRole: 'admin', capability: 'invitations.send', + contentDelivery: 'none', workspaceApiKey: 'deny', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], diff --git a/apps/sim/lib/invitations/application/operations.ts b/apps/sim/lib/invitations/application/operations.ts index 435e6474ba4..d72e3c74c41 100644 --- a/apps/sim/lib/invitations/application/operations.ts +++ b/apps/sim/lib/invitations/application/operations.ts @@ -60,6 +60,7 @@ export const workspaceInvitationOperations = { id: 'workspace_invitations.send_batch', minimumRole: 'admin', capability: 'invitations.send', + contentDelivery: 'none', workspaceApiKey: 'deny', principalKinds: ['session', 'personal_api_key', 'oauth_access_token', 'delegated'], delegatedServices: ['copilot'], diff --git a/apps/sim/lib/invitations/application/send-invitation-batch.ts b/apps/sim/lib/invitations/application/send-invitation-batch.ts index fde30ee9262..9e2d7ca5588 100644 --- a/apps/sim/lib/invitations/application/send-invitation-batch.ts +++ b/apps/sim/lib/invitations/application/send-invitation-batch.ts @@ -341,6 +341,7 @@ export const sendWorkspaceInvitationBatch = defineAuthorizedWorkspaceUseCase({ const organizationInvitationSendOperation = defineOrganizationOperation({ id: 'organization_invitations.send_batch', + contentDelivery: 'none', minimumRole: 'admin', capability: 'invitations.send', principalKinds: ['session', 'organization_delegated'], diff --git a/apps/sim/lib/knowledge/__integration__/content-delivery.integration.ts b/apps/sim/lib/knowledge/__integration__/content-delivery.integration.ts new file mode 100644 index 00000000000..33f44ead783 --- /dev/null +++ b/apps/sim/lib/knowledge/__integration__/content-delivery.integration.ts @@ -0,0 +1,604 @@ +/** Real PostgreSQL content admission, pagination, captured metadata, and streamed ZIP readback. */ +import { mkdtempSync } from 'node:fs' +import { rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import path from 'node:path' +import * as audit from '@sim/audit' +import { db } from '@sim/db' +import { + auditLog, + document, + embedding, + knowledgeBase, + organization, + user, + workspace, +} from '@sim/db/schema' +import { deleteWorkspaceFixture } from '@sim/db/testing/workspace-fixtures' +import { sha256Hex } from '@sim/security/hash' +import { generateId } from '@sim/utils/id' +import { isRecordLike } from '@sim/utils/object' +import { eq, inArray } from 'drizzle-orm' +import { PostgresJsPreparedQuery } from 'drizzle-orm/postgres-js/session' +import JSZip from 'jszip' +import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from 'vitest' + +const fixtureStorage = vi.hoisted(() => ({ root: '' })) +vi.mock('@/lib/uploads/core/setup.server', () => ({ + get UPLOAD_DIR_SERVER() { + return fixtureStorage.root + }, +})) + +import { observeContentDelivery } from '@/lib/core/application' +import { encryptSecret } from '@/lib/core/security/encryption' +import type { DurableSecretProvenance } from '@/lib/execution/durable-secret-provenance' +import { + createKnowledgeAclFixtureIds, + seedKnowledgeAclFixture, +} from '@/lib/knowledge/__integration__/seed-source-access-fixture' +import { admitKnowledgeChunks } from '@/lib/knowledge/application/admit-knowledge-content' +import { KNOWLEDGE_DELEGATION_AUDIENCE } from '@/lib/knowledge/application/authorization' +import { listKnowledgeChunks } from '@/lib/knowledge/application/chunks' +import * as knowledgeContexts from '@/lib/knowledge/application/contexts' +import { listKnowledgeDocuments } from '@/lib/knowledge/application/documents' +import { + exportKnowledgeBase, + type KnowledgeBaseExportBundle, +} from '@/lib/knowledge/application/exports' +import { readOriginalKnowledgeDocument } from '@/lib/knowledge/application/read-original-document' +import { + readDetailedKnowledgeTagUsage, + readKnowledgeTagUsage, +} from '@/lib/knowledge/application/tags' +import { + bindKnowledgeDocumentFieldSecretProvenance, + classifyKnowledgeDocumentMetadata, + createKnowledgeDocumentSourceValue, + replaceKnowledgeDocumentSecretProvenanceInTx, + replaceKnowledgeEmbeddingSecretProvenanceInTx, +} from '@/lib/knowledge/secret-provenance' +import { buildKnowledgeBundleArchive } from '@/lib/knowledge/transfer/export-archive' +import { embeddingVectorValues } from '@/lib/knowledge/vector-columns' +import { createCopilotChatPrincipal } from '@/lib/mothership/auth/application-delegation' +import { createRegistryContentGate } from '@/lib/mothership/content-delivery/registry-gate' +import { uploadFile } from '@/lib/uploads/core/storage-service' +import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry' + +const SECRET = 'fixture-knowledge-token-41875' +const ids = createKnowledgeAclFixtureIds() +const principal = createCopilotChatPrincipal( + { userId: ids.aliceId, workspaceId: ids.workspaceId }, + KNOWLEDGE_DELEGATION_AUDIENCE +) +const session = { kind: 'session' as const, userId: ids.aliceId, sessionId: generateId() } +let exact: DurableSecretProvenance +let submittedAudits = 0 + +function inChat(read: () => T) { + return observeContentDelivery( + createRegistryContentGate({ + registry: new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + actorUserId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + read + ) +} + +async function seedDocument( + options: { + name?: string + content?: string + provenance?: DurableSecretProvenance + original?: boolean + } = {} +) { + const content = options.content ?? 'page public content' + const id = generateId() + const [row] = await db + .insert(document) + .values({ + id, + knowledgeBaseId: ids.knowledgeBaseId, + filename: options.name ?? 'page.txt', + fileUrl: + options.original === false + ? '' + : `data:text/plain;base64,${Buffer.from(content).toString('base64')}`, + fileSize: Buffer.byteLength(content), + mimeType: 'text/plain', + processingStatus: 'completed', + chunkCount: 1, + tag1: 'public', + acl: ['ws'], + aclRequirements: [], + aclVerifiedAt: new Date(), + }) + .returning() + const provenance = options.provenance + if (provenance) + await db.transaction((tx) => + replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + id, + createKnowledgeDocumentSourceValue(row), + provenance + ) + ) + const chunkId = await seedChunk(id, 0, content) + return { id, chunkId, content } +} + +async function seedChunk( + documentId: string, + index: number, + content: string, + provenance?: DurableSecretProvenance +) { + const id = generateId() + await db.insert(embedding).values({ + id, + documentId, + knowledgeBaseId: ids.knowledgeBaseId, + chunkIndex: index, + ...embeddingVectorValues(1536, [1, ...Array(1535).fill(0)]), + chunkHash: sha256Hex(content), + content, + contentLength: content.length, + tokenCount: 4, + startOffset: 0, + endOffset: content.length, + tag1: 'public', + }) + if (provenance) + await db.transaction((tx) => + replaceKnowledgeEmbeddingSecretProvenanceInTx(tx, id, content, provenance) + ) + return id +} + +async function exportedBundle(human = false) { + return inChat(() => + exportKnowledgeBase.execute({ + principal: human ? session : principal, + input: { knowledgeBaseId: ids.knowledgeBaseId, vectors: false }, + }) + ) +} + +async function archiveBytes(bundle: KnowledgeBaseExportBundle) { + const buffers: Buffer[] = [] + for await (const chunk of buildKnowledgeBundleArchive(bundle)) buffers.push(Buffer.from(chunk)) + return Buffer.concat(buffers) +} + +async function drainAudits() { + await vi.waitFor(async () => + expect( + await db + .select({ id: auditLog.id }) + .from(auditLog) + .where(eq(auditLog.workspaceId, ids.workspaceId)) + ).toHaveLength(submittedAudits) + ) +} + +beforeAll(async () => { + fixtureStorage.root = mkdtempSync(path.join(tmpdir(), 'sim-knowledge-delivery-')) + await seedKnowledgeAclFixture(ids) + const record = audit.recordAudit + vi.spyOn(audit, 'recordAudit').mockImplementation((entry) => { + if (entry.workspaceId === ids.workspaceId) submittedAudits += 1 + record(entry) + }) + const recordBatch = audit.recordAuditBatch + vi.spyOn(audit, 'recordAuditBatch').mockImplementation((entries) => { + submittedAudits += entries.filter((entry) => entry.workspaceId === ids.workspaceId).length + recordBatch(entries) + }) + exact = { + status: 'exact', + entries: [ + { + encryptedValue: (await encryptSecret(SECRET)).encrypted, + sourceUserId: ids.aliceId, + sourceWorkspaceId: ids.workspaceId, + name: 'FIXTURE_TOKEN', + }, + ], + } +}) +afterEach(async () => { + await drainAudits() + await db.delete(document).where(eq(document.knowledgeBaseId, ids.knowledgeBaseId)) +}) +afterAll(async () => { + await drainAudits() + vi.restoreAllMocks() + await db.delete(auditLog).where(eq(auditLog.workspaceId, ids.workspaceId)) + await db.delete(knowledgeBase).where(eq(knowledgeBase.id, ids.knowledgeBaseId)) + await deleteWorkspaceFixture(db, eq(workspace.id, ids.workspaceId)) + await db.delete(organization).where(eq(organization.id, ids.organizationId)) + await db.delete(user).where(inArray(user.id, [ids.aliceId, ids.bobId])) + await rm(fixtureStorage.root, { recursive: true, force: true }) + await db.$client.end() +}) + +describe('knowledge content delivery', () => { + it('classifies a mixed metadata page without materializing multiple inline originals together', async () => { + const content = 'inline source content '.repeat(8192) + const legacy = await seedDocument({ content }) + const clean = await seedDocument({ content, provenance: { status: 'exact', entries: [] } }) + const unknown = await seedDocument({ content, provenance: { status: 'unknown' } }) + const tracked = await seedDocument({ content }) + const [trackedRow] = await db + .update(document) + .set({ tag1: SECRET }) + .where(eq(document.id, tracked.id)) + .returning() + await db.transaction((tx) => + replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + tracked.id, + createKnowledgeDocumentSourceValue(trackedRow), + bindKnowledgeDocumentFieldSecretProvenance(exact, 'tag1', SECRET) + ) + ) + const originalBytes = Buffer.byteLength( + `data:text/plain;base64,${Buffer.from(content).toString('base64')}` + ) + let peakOriginalBytes = 0 + const execute = PostgresJsPreparedQuery.prototype.execute + const measure = vi + .spyOn(PostgresJsPreparedQuery.prototype, 'execute') + .mockImplementation(async function (this: typeof PostgresJsPreparedQuery.prototype, values) { + const rows = await execute.call(this, values) + if (Array.isArray(rows)) { + let bytes = 0 + for (const row of rows) { + if (!isRecordLike(row)) continue + const value = row.fileUrl + if (typeof value === 'string' && value.startsWith('data:')) + bytes += Buffer.byteLength(value) + } + peakOriginalBytes = Math.max(peakOriginalBytes, bytes) + } + return rows + }) + try { + const result = await classifyKnowledgeDocumentMetadata([ + legacy.id, + clean.id, + unknown.id, + tracked.id, + ]) + expect(result.get(legacy.id)).toEqual({ status: 'exact', entries: [] }) + expect(result.get(clean.id)).toEqual({ status: 'exact', entries: [] }) + expect(result.get(unknown.id)).toEqual({ status: 'unknown' }) + expect(result.get(tracked.id)).toMatchObject({ + status: 'exact', + entries: [{ name: 'FIXTURE_TOKEN' }], + }) + expect(peakOriginalBytes).toBeGreaterThan(0) + expect(peakOriginalBytes).toBeLessThanOrEqual(originalBytes) + } finally { + measure.mockRestore() + } + }) + + it('streams chunk pages without reloading the tracked inline original for each page', async () => { + const content = 'inline original content '.repeat(4096) + const doc = await seedDocument({ content, provenance: { status: 'exact', entries: [] } }) + await db.insert(embedding).values( + Array.from({ length: 100 }, (_, index) => { + const text = `public exported chunk ${index + 1}` + return { + id: generateId(), + documentId: doc.id, + knowledgeBaseId: ids.knowledgeBaseId, + chunkIndex: index + 1, + ...embeddingVectorValues(1536, [1, ...Array(1535).fill(0)]), + chunkHash: sha256Hex(text), + content: text, + contentLength: text.length, + tokenCount: 4, + startOffset: 0, + endOffset: text.length, + tag1: 'public', + } + }) + ) + const bundle = await inChat(() => + exportKnowledgeBase.execute({ + principal, + input: { knowledgeBaseId: ids.knowledgeBaseId, vectors: true }, + }) + ) + const originalBytes = Buffer.byteLength( + `data:text/plain;base64,${Buffer.from(content).toString('base64')}` + ) + let loadedOriginalBytes = 0 + const execute = PostgresJsPreparedQuery.prototype.execute + const measure = vi + .spyOn(PostgresJsPreparedQuery.prototype, 'execute') + .mockImplementation(async function (this: typeof PostgresJsPreparedQuery.prototype, values) { + const rows = await execute.call(this, values) + if (Array.isArray(rows)) { + for (const row of rows) { + if (!isRecordLike(row)) continue + const value = row.fileUrl + if (typeof value === 'string' && value.startsWith('data:')) + loadedOriginalBytes += Buffer.byteLength(value) + } + } + return rows + }) + try { + const exported: string[] = [] + for await (const chunk of bundle.chunks(doc.id)) exported.push(chunk.content) + expect(exported).toHaveLength(101) + expect(exported[0]).toBe(content) + expect(exported.at(-1)).toBe('public exported chunk 100') + expect(loadedOriginalBytes).toBeGreaterThan(0) + expect(loadedOriginalBytes).toBeLessThanOrEqual(originalBytes) + } finally { + measure.mockRestore() + } + }) + + it('keeps stored search totals and continuation while withholding only unsafe documents', async () => { + await seedDocument({ name: 'page-a.txt' }) + await seedDocument({ name: 'page-b.txt', provenance: { status: 'unknown' } }) + await seedDocument({ name: 'page-c.txt' }) + const result = await inChat(() => + listKnowledgeDocuments.execute({ + principal, + input: { + knowledgeBaseId: ids.knowledgeBaseId, + search: 'page', + limit: 2, + sortBy: 'filename', + sortOrder: 'asc', + }, + }) + ) + expect(result.documents.map((row) => row.filename)).toEqual(['page-a.txt']) + expect(result.pagination).toMatchObject({ total: 3, hasMore: true }) + }) + + it('keeps chunk search totals and its next page when one matching chunk is withheld', async () => { + const doc = await seedDocument() + await seedChunk(doc.id, 1, 'page withheld content', { status: 'unknown' }) + await seedChunk(doc.id, 2, 'page final content') + const result = await inChat(() => + listKnowledgeChunks.execute({ + principal, + input: { + knowledgeBaseId: ids.knowledgeBaseId, + documentId: doc.id, + search: 'page', + limit: 2, + }, + }) + ) + expect(result.chunks.map((row) => row.content)).toEqual(['page public content']) + expect(result.pagination).toMatchObject({ total: 3, hasMore: true }) + expect(result.nextCursorKeys).not.toBeNull() + }) + + it('returns ordinary aggregate counts while admitting detailed tag metadata per document', async () => { + const clean = await seedDocument() + await seedDocument({ provenance: { status: 'unknown' } }) + const stats = await inChat(() => + readKnowledgeTagUsage.execute({ principal, input: { knowledgeBaseId: ids.knowledgeBaseId } }) + ) + expect(stats.usage.find((row) => row.tagSlot === 'tag1')).toMatchObject({ + documentCount: 2, + chunkCount: 2, + }) + const detail = await inChat(() => + readDetailedKnowledgeTagUsage.execute({ + principal, + input: { knowledgeBaseId: ids.knowledgeBaseId }, + }) + ) + expect(detail.usage[0].documents.map((row) => row.id)).toEqual([clean.id]) + }) + + it.each(['tag1', 'filename'] as const)( + 'binds captured chunk %s metadata to the row classified at delivery', + async (field) => { + const doc = await seedDocument() + const captured = { + id: doc.chunkId, + documentId: doc.id, + content: doc.content, + value: { content: doc.content, [field]: SECRET }, + } + await db + .update(document) + .set({ [field]: SECRET }) + .where(eq(document.id, doc.id)) + const [secretRow] = await db.select().from(document).where(eq(document.id, doc.id)) + await db.transaction((tx) => + replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + doc.id, + createKnowledgeDocumentSourceValue(secretRow), + bindKnowledgeDocumentFieldSecretProvenance(exact, field, SECRET) + ) + ) + const gate = createRegistryContentGate({ + registry: new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + actorUserId: ids.aliceId, + workspaceId: ids.workspaceId, + }) + expect( + await observeContentDelivery(gate, () => admitKnowledgeChunks(principal, [captured])) + ).toEqual(new Set([doc.chunkId])) + expect(gate.project?.(SECRET)).toBe('{{FIXTURE_TOKEN}}') + await db.transaction(async (tx) => { + const [current] = await tx + .update(document) + .set({ [field]: 'public' }) + .where(eq(document.id, doc.id)) + .returning() + await replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + doc.id, + createKnowledgeDocumentSourceValue(current), + { status: 'exact', entries: [] } + ) + }) + expect(await inChat(() => admitKnowledgeChunks(principal, [captured]))).toEqual(new Set()) + } + ) + + it('binds an original read to the captured storage key across a concurrent replacement', async () => { + const doc = await seedDocument({ original: false, content: SECRET, provenance: exact }) + const oldKey = `kb/${generateId()}.txt` + const newKey = `kb/${generateId()}.txt` + for (const [key, content] of [ + [oldKey, SECRET], + [newKey, 'replacement public content'], + ]) { + await uploadFile({ + file: Buffer.from(content), + fileName: 'fixture.txt', + contentType: 'text/plain', + context: 'knowledge-base', + customKey: key, + preserveKey: true, + persistMetadata: false, + }) + } + await db.update(document).set({ storageKey: oldKey }).where(eq(document.id, doc.id)) + const input = { + knowledgeBaseId: ids.knowledgeBaseId, + documentId: doc.id, + assertedWorkspaceId: ids.workspaceId, + } + const matched = await inChat(() => readOriginalKnowledgeDocument.execute({ principal, input })) + expect(matched.buffer?.toString()).toBe(SECRET) + expect(matched.hasRecordedSecrets).toBe(true) + const resolve = knowledgeContexts.resolveCanonicalActiveKnowledgeDocumentContext + let replaced = false + const interleave = vi + .spyOn(knowledgeContexts, 'resolveCanonicalActiveKnowledgeDocumentContext') + .mockImplementation(async (...args) => { + const context = await resolve(...args) + await db.transaction(async (tx) => { + const [row] = await tx + .update(document) + .set({ storageKey: newKey }) + .where(eq(document.id, doc.id)) + .returning() + await replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + doc.id, + createKnowledgeDocumentSourceValue(row), + { status: 'exact', entries: [] } + ) + }) + replaced = true + return context + }) + try { + await expect( + inChat(() => readOriginalKnowledgeDocument.execute({ principal, input })) + ).rejects.toThrow(/provenance|model/i) + expect(replaced).toBe(true) + } finally { + interleave.mockRestore() + } + const current = await inChat(() => readOriginalKnowledgeDocument.execute({ principal, input })) + expect(current.buffer?.toString()).toBe('replacement public content') + expect(current.hasRecordedSecrets).toBe(false) + }) + + it('exports clean and legacy originals with chunks through the existing ZIP stream', async () => { + const docs = [ + await seedDocument(), + await seedDocument({ name: 'tracked.txt', provenance: { status: 'exact', entries: [] } }), + ] + const bundle = await exportedBundle() + const zip = await JSZip.loadAsync(await archiveBytes(bundle)) + for (const doc of docs) { + const original = Object.values(zip.files).find( + (file) => file.name.startsWith(`files/${doc.id}/`) && !file.dir + ) + expect(await original?.async('string')).toBe(doc.content) + const lines = await zip.file(`chunks/${doc.id}.ndjson`)?.async('string') + expect(JSON.parse(lines?.trim() ?? '{}').content).toBe(doc.content) + } + expect(zip.file('manifest.json')).not.toBeNull() + }) + + it.each(['unknown', 'exact', 'corrupt'] as const)( + 'refuses %s chunk bytes introduced after bundle creation, even outside the original gate context', + async (kind) => { + const doc = await seedDocument({ original: false }) + const bundle = await exportedBundle() + await db.transaction(async (tx) => { + await tx + .update(embedding) + .set({ content: SECRET, chunkHash: sha256Hex(SECRET) }) + .where(eq(embedding.id, doc.chunkId)) + await replaceKnowledgeEmbeddingSecretProvenanceInTx( + tx, + doc.chunkId, + kind === 'corrupt' ? 'previous content' : SECRET, + kind === 'exact' + ? exact + : kind === 'unknown' + ? { status: 'unknown' } + : { status: 'exact', entries: [] } + ) + }) + await expect(archiveBytes(bundle)).rejects.toThrow(/provenance|model/i) + const emitted: string[] = [] + await expect( + (async () => { + for await (const chunk of bundle.chunks(doc.id)) emitted.push(chunk.content) + })() + ).rejects.toThrow(/provenance|model/i) + expect(emitted).toEqual([]) + } + ) + + it('refuses an inline original replaced after bundle creation without exposing new bytes', async () => { + const doc = await seedDocument() + const bundle = await exportedBundle() + await db.transaction(async (tx) => { + const [row] = await tx + .update(document) + .set({ fileUrl: `data:text/plain;base64,${Buffer.from(SECRET).toString('base64')}` }) + .where(eq(document.id, doc.id)) + .returning() + await replaceKnowledgeDocumentSecretProvenanceInTx( + tx, + doc.id, + createKnowledgeDocumentSourceValue(row), + exact + ) + }) + await expect(archiveBytes(bundle)).rejects.toThrow(/provenance|model/i) + }) + + it('preserves authorized human export of the original and chunks', async () => { + const doc = await seedDocument({ content: SECRET, provenance: { status: 'unknown' } }) + const zip = await JSZip.loadAsync(await archiveBytes(await exportedBundle(true))) + expect(await zip.file(`chunks/${doc.id}.ndjson`)?.async('string')).toContain(SECRET) + const original = Object.values(zip.files).find( + (file) => file.name.startsWith(`files/${doc.id}/`) && !file.dir + ) + expect(await original?.async('string')).toBe(SECRET) + }) +}) diff --git a/apps/sim/lib/knowledge/__integration__/execution-archive-provenance.integration.ts b/apps/sim/lib/knowledge/__integration__/execution-archive-provenance.integration.ts index 3f0fd094a10..fbe7e946692 100644 --- a/apps/sim/lib/knowledge/__integration__/execution-archive-provenance.integration.ts +++ b/apps/sim/lib/knowledge/__integration__/execution-archive-provenance.integration.ts @@ -24,6 +24,7 @@ import JSZip from 'jszip' import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' const fixtureStorage = vi.hoisted(() => ({ root: '' })) +const embeddingInputs = vi.hoisted(() => new Map()) const sandboxEnvironment = vi.hoisted(() => { const fixture = { SANDBOX_PROVIDER: 'e2b', @@ -45,18 +46,26 @@ vi.mock('@/lib/uploads/core/setup.server', () => ({ vi.mock('@/lib/embeddings', async () => ({ ...(await import('@/lib/embeddings/client')), assertKnowledgeEmbeddingCapacity: async () => {}, - embedKnowledge: async (texts: string[]) => ({ - embeddings: texts.map(() => [1, ...Array(1535).fill(0)]), - totalTokens: texts.length, - billableTokens: 0, - isBYOK: true, - modelName: 'text-embedding-3-small', - pricingId: 'text-embedding-3-small', - }), + embedKnowledge: async (texts: string[], options: { workspaceId?: string | null }) => { + if (options.workspaceId) { + const inputs = embeddingInputs.get(options.workspaceId) ?? [] + inputs.push(...texts) + embeddingInputs.set(options.workspaceId, inputs) + } + return { + embeddings: texts.map(() => [1, ...Array(1535).fill(0)]), + totalTokens: texts.length, + billableTokens: 0, + isBYOK: true, + modelName: 'text-embedding-3-small', + pricingId: 'text-embedding-3-small', + } + }, })) import { functionExecuteBodySchema } from '@/lib/api/contracts' import { fileManageDecompressBodySchema } from '@/lib/api/contracts/tools/file' +import { observeContentDelivery } from '@/lib/core/application' import { processOutboxEventById } from '@/lib/core/outbox/service' import { decryptSecret, encryptSecret } from '@/lib/core/security/encryption' import { isUserFile } from '@/lib/core/utils/user-file' @@ -76,16 +85,20 @@ import { seedKnowledgeAclFixture, } from '@/lib/knowledge/__integration__/seed-source-access-fixture' import { addWorkspaceFilesToKnowledgeBase } from '@/lib/knowledge/application/add-workspace-files' +import { KNOWLEDGE_DELEGATION_AUDIENCE } from '@/lib/knowledge/application/authorization' import { listKnowledgeChunks } from '@/lib/knowledge/application/chunks' import { searchKnowledge } from '@/lib/knowledge/application/search' import { KNOWLEDGE_DOCUMENT_PROCESSING_OUTBOX_EVENT } from '@/lib/knowledge/documents/processing-events' import { knowledgeDocumentProcessingOutboxHandlers } from '@/lib/knowledge/documents/processing-outbox-handler' import { createSingleDocument } from '@/lib/knowledge/documents/service' import { loadKnowledgeDocumentSecretRegistry } from '@/lib/knowledge/secret-provenance' +import { createCopilotChatPrincipal } from '@/lib/mothership/auth/application-delegation' +import { createRegistryContentGate } from '@/lib/mothership/content-delivery/registry-gate' import { uploadExecutionFile } from '@/lib/uploads/contexts/execution/execution-file-manager' import { deleteWorkspaceFile, getWorkspaceFile, + uploadWorkspaceFile, } from '@/lib/uploads/contexts/workspace/workspace-file-manager' import { filterModelSafeWorkspaceFileAttachments, @@ -190,13 +203,32 @@ async function extract(ids: Fixture, archive: UserFile) { return { child, record, identity, publicMetadata: JSON.stringify({ archive, body }) } } -async function assertBlockedConsumers(ids: Fixture, source: Awaited>) { +/** + * Opaque delivery is refused either way. A knowledge base refuses an unknown source but accepts + * one with recorded secrets, whose entries its chunks then carry (redacted wherever they are read). + */ +async function assertBlockedConsumers( + ids: Fixture, + source: Awaited>, + knowledge: 'refused' | 'accepted' = 'refused' +) { expect(await isOpaqueWorkspaceFileEgressSafe(ids.workspaceId, source.identity)).toBe(false) const imported = await addWorkspaceFilesToKnowledgeBase.execute({ principal: sessionPrincipal(ids), input: { knowledgeBaseId: ids.knowledgeBaseId, fileReferences: [source.child.id] }, }) - expect(imported).toMatchObject({ added: [], failed: [source.child.id] }) + if (knowledge === 'refused') { + expect(imported).toMatchObject({ added: [], failed: [source.child.id] }) + } else { + expect(imported).toMatchObject({ failed: [] }) + expect(imported.added).toHaveLength(1) + /** The accepted document queued its processing; that work is removed with the fixture. */ + const events = await db + .select({ id: outboxEvent.id }) + .from(outboxEvent) + .where(sql`${outboxEvent.payload}::jsonb ->> 'documentId' = ${imported.added[0].documentId}`) + trackedEventIds.push(...events.map((event) => event.id)) + } } beforeAll(() => { @@ -696,6 +728,119 @@ describe('Function export provenance in PostgreSQL', () => { }) describe('execution archive durable provenance', () => { + it.each([ + { + name: 'whitespace-normalized secret', + secret: ' fixture-live-key-82367 ', + fragments: ['fixture-live-key-82367'], + masked: true, + }, + { + name: 'secret crossing chunk boundaries', + secret: `left-fixture-secret-${'Q'.repeat(180)}-right-fixture-secret`, + fragments: ['left-fixture-secret', 'QQQQQQQQ', 'right-fixture-secret'], + masked: true, + }, + { name: 'short secret cutoff', secret: 'T0k3n42', fragments: [], masked: false }, + ])( + 'indexes $name without changing original bytes or exposing derived fragments', + async (testCase) => { + const ids = await seed() + await db + .update(knowledgeBase) + .set({ chunkingConfig: { maxSize: 32, minSize: 1, overlap: 0 } }) + .where(eq(knowledgeBase.id, ids.knowledgeBaseId)) + const { encrypted } = await encryptSecret(testCase.secret) + const content = `Public introduction.\n${testCase.secret}\nPublic conclusion.` + const source = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from(content), + 'source.txt', + 'text/plain', + { + notifyWorkspaceChange: false, + secretProvenance: { + status: 'exact', + entries: [ + { + name: 'FIXTURE_SECRET', + encryptedValue: encrypted, + sourceUserId: ids.aliceId, + sourceWorkspaceId: ids.workspaceId, + }, + ], + }, + } + ) + const imported = await addWorkspaceFilesToKnowledgeBase.execute({ + principal: sessionPrincipal(ids), + input: { knowledgeBaseId: ids.knowledgeBaseId, fileReferences: [source.id] }, + }) + expect(imported.failed).toEqual([]) + expect(imported.added).toHaveLength(1) + const documentId = imported.added[0].documentId + const events = await db + .select() + .from(outboxEvent) + .where(sql`${outboxEvent.payload}::jsonb ->> 'documentId' = ${documentId}`) + trackedEventIds.push(...events.map((event) => event.id)) + const dispatch = events.find( + (event) => event.eventType === KNOWLEDGE_DOCUMENT_PROCESSING_OUTBOX_EVENT + ) + if (!dispatch) throw new Error('Knowledge import did not queue document processing') + await processOutboxEventById(dispatch.id, knowledgeDocumentProcessingOutboxHandlers) + const [indexed] = await db.select().from(document).where(eq(document.id, documentId)) + expect(indexed.processingStatus, indexed.processingError ?? undefined).toBe('completed') + if (!indexed.storageKey) throw new Error('Knowledge import did not retain its original') + expect( + (await downloadFile({ key: indexed.storageKey, context: 'knowledge-base' })).toString() + ).toBe(content) + const stored = await listKnowledgeChunks.execute({ + principal: sessionPrincipal(ids), + input: { knowledgeBaseId: ids.knowledgeBaseId, documentId }, + }) + const readbackRegistry = new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }) + const gate = createRegistryContentGate({ + registry: readbackRegistry, + actorUserId: ids.aliceId, + workspaceId: ids.workspaceId, + }) + const readback = await observeContentDelivery(gate, () => + listKnowledgeChunks.execute({ + principal: createCopilotChatPrincipal( + { userId: ids.aliceId, workspaceId: ids.workspaceId }, + KNOWLEDGE_DELEGATION_AUDIENCE + ), + input: { knowledgeBaseId: ids.knowledgeBaseId, documentId }, + }) + ) + expect(readback.chunks.length).toBeGreaterThan(0) + const projected = projectResolvedSecretModelContent(readback, readbackRegistry) + expect(projected.safe).toBe(true) + if (!projected.safe) throw new Error('Indexed chunk readback was withheld') + const outputs = [ + (embeddingInputs.get(ids.workspaceId) ?? []).join('\n'), + stored.chunks.map((chunk) => chunk.content).join('\n'), + JSON.stringify(projected.value), + ] + for (const output of outputs) { + expect(output).toContain('Public introduction.') + expect(output).toContain('Public conclusion.') + if (testCase.masked) { + expect(output).toContain('{{FIXTURE_SECRET}}') + for (const fragment of testCase.fragments) expect(output).not.toContain(fragment) + } else { + expect(output).toContain(testCase.secret) + expect(output).not.toContain('{{FIXTURE_SECRET}}') + } + } + } + ) + it.each(['exact', 'unrecorded'] as const)( 'carries %s lineage through extraction and delayed KB indexing/search', async (status) => { @@ -827,7 +972,7 @@ describe('execution archive durable provenance', () => { expect(projected.safe).toBe(true) if (!projected.safe) throw new Error('Known extracted lineage withheld runtime readback') expect(projected.value).toBe('name,description\nOrion,{{FIXTURE_SECRET}}\n') - await assertBlockedConsumers(ids, source) + await assertBlockedConsumers(ids, source, 'accepted') }) it('preserves compatibility for execution files created before provenance stamping', async () => { diff --git a/apps/sim/lib/knowledge/__integration__/upload-read-provenance.integration.ts b/apps/sim/lib/knowledge/__integration__/upload-read-provenance.integration.ts index 72b0687f2cb..d44f0fc0267 100644 --- a/apps/sim/lib/knowledge/__integration__/upload-read-provenance.integration.ts +++ b/apps/sim/lib/knowledge/__integration__/upload-read-provenance.integration.ts @@ -101,7 +101,6 @@ async function readUpload(ids: Awaited>) { workspaceId: ids.workspaceId, chatId: ids.chatId, reference: `uploads/${ids.name}`, - includeSecretProvenance: true, }, }) return { diff --git a/apps/sim/lib/knowledge/application/add-workspace-files.ts b/apps/sim/lib/knowledge/application/add-workspace-files.ts index 241d463f469..36ac1b65c9a 100644 --- a/apps/sim/lib/knowledge/application/add-workspace-files.ts +++ b/apps/sim/lib/knowledge/application/add-workspace-files.ts @@ -37,6 +37,8 @@ import { } from '@/lib/uploads/contexts/workspace/workspace-file-manager' import { getBoundWorkspaceFileSecretProvenance, + mergeWorkspaceFileSecretProvenance, + type WorkspaceFileSecretProvenance, type WorkspaceFileSecretProvenanceIdentity, } from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance' import { @@ -81,7 +83,7 @@ interface AddWorkspaceFilesContext extends ActiveKnowledgeBaseContext { interface PreparedWorkspaceFile { reference: string file: WorkspaceFileRecord - unrecordedProvenance: boolean + provenance: WorkspaceFileSecretProvenance } async function prepareWorkspaceFile( @@ -111,7 +113,7 @@ async function prepareWorkspaceFile( const fileTypeError = validateFileType(file.name, file.type) if (fileTypeError) throw new OrchestrationError('validation', fileTypeError.message) - const unrecordedProvenance = await assertImportProvenance(context.workspaceId, { + const provenance = await readImportProvenance(context.workspaceId, { fileId: file.id, key: file.key, context: file.storageContext ?? 'workspace', @@ -121,25 +123,27 @@ async function prepareWorkspaceFile( return { reference, file, - unrecordedProvenance, + provenance, } } -async function assertImportProvenance( +/** + * The provenance a source file brings into the document. Its secret entries carry over to the + * document's content, and from there to each chunk that holds them, so reads redact them; only a + * source whose provenance is unknown is refused, since nothing could be carried for it. + */ +async function readImportProvenance( workspaceId: string, identity: WorkspaceFileSecretProvenanceIdentity -): Promise { +): Promise { const provenance = await getBoundWorkspaceFileSecretProvenance(workspaceId, identity) - if ( - provenance.status === 'unknown' || - (provenance.status === 'exact' && provenance.entries.length > 0) - ) { + if (provenance.status === 'unknown') { throw new OrchestrationError( 'validation', 'Workspace file secret provenance prevents knowledge ingestion' ) } - return provenance.status === 'unrecorded' + return provenance } export const addWorkspaceFilesToKnowledgeBase = defineAuthorizedKnowledgeUseCase({ @@ -221,7 +225,7 @@ export const addWorkspaceFilesToKnowledgeBase = defineAuthorizedKnowledgeUseCase const requestId = generateRequestId() const current = await prepareWorkspaceFile(principal, context, candidate.file.id, 'id') const unrecordedSources = new Set() - if (current.unrecordedProvenance) unrecordedSources.add(current.file.id) + const sourceProvenance: WorkspaceFileSecretProvenance[] = [] const readSignal = AbortSignal.timeout(SOURCE_READ_TIMEOUT_MS) const signal = input.cancellationSignal ? AbortSignal.any([readSignal, input.cancellationSignal]) @@ -244,9 +248,9 @@ export const addWorkspaceFilesToKnowledgeBase = defineAuthorizedKnowledgeUseCase } for (const identity of contributors) { signal.throwIfAborted() - if (await assertImportProvenance(context.workspaceId, identity)) { - unrecordedSources.add(identity.fileId) - } + const provenance = await readImportProvenance(context.workspaceId, identity) + if (provenance.status === 'unrecorded') unrecordedSources.add(identity.fileId) + sourceProvenance.push(provenance) } const documentId = generateId() const storedFile = await uploadKnowledgeArtifact({ @@ -274,7 +278,17 @@ export const addWorkspaceFilesToKnowledgeBase = defineAuthorizedKnowledgeUseCase ) { throw new OrchestrationError('conflict', 'Workspace file changed during knowledge import') } - if (finalized.unrecordedProvenance) unrecordedSources.add(finalized.file.id) + if (finalized.provenance.status === 'unrecorded') unrecordedSources.add(finalized.file.id) + const content = mergeWorkspaceFileSecretProvenance( + finalized.provenance, + ...sourceProvenance + ) + if (content.status === 'unknown') { + throw new OrchestrationError( + 'validation', + 'Workspace file secret provenance prevents knowledge ingestion' + ) + } input.cancellationSignal?.throwIfAborted() const document = await createSingleDocument( { @@ -289,7 +303,7 @@ export const addWorkspaceFilesToKnowledgeBase = defineAuthorizedKnowledgeUseCase documentId, { filename: { status: 'exact', entries: [] }, - content: { status: 'exact', entries: [] }, + content: content.status === 'exact' ? content : { status: 'exact', entries: [] }, tags: [], }, { diff --git a/apps/sim/lib/knowledge/application/admit-knowledge-content.ts b/apps/sim/lib/knowledge/application/admit-knowledge-content.ts new file mode 100644 index 00000000000..736322e5cdf --- /dev/null +++ b/apps/sim/lib/knowledge/application/admit-knowledge-content.ts @@ -0,0 +1,91 @@ +import type { Principal } from '@sim/auth/principal' +import { admitEach, type ContentSourceKind } from '@/lib/core/application' +import { + classifyKnowledgeChunks, + classifyKnowledgeDocumentMetadata, + type KnowledgeDocumentSnapshot, + knowledgeDocumentSnapshot, +} from '@/lib/knowledge/secret-provenance' + +interface KnowledgeAdmissionOptions { + /** The items are what a committed write returns; see `admitWrittenContent`. */ + written?: boolean +} + +const writtenAs = (options: KnowledgeAdmissionOptions, kind: ContentSourceKind) => + options.written ? { written: kind } : {} + +/** + * Admits each document whose rendered metadata a result returns, on its own, and returns the ids + * admitted. `value` is what the result shows of the document, so only secrets it contains are + * imported; `snapshot` names the stored fields it shows, when they are not its own keys. A caller + * no gate observes gets every id back without a provenance read. + */ +export async function admitKnowledgeDocuments( + principal: Principal, + documents: readonly { id: string; value: unknown; snapshot?: KnowledgeDocumentSnapshot }[], + options: KnowledgeAdmissionOptions = {} +): Promise> { + const admitted = await admitEach( + principal, + documents, + async (items) => { + /** Each classification is bound to the metadata the item shows; see the classifier. */ + const snapshots = new Map() + for (const item of items) { + const snapshot = item.snapshot ?? knowledgeDocumentSnapshot(item.value) + if (snapshot) snapshots.set(item.id, [...(snapshots.get(item.id) ?? []), snapshot]) + } + const provenance = await classifyKnowledgeDocumentMetadata( + items.map((item) => item.id), + snapshots + ) + return items.map((item) => ({ + kind: 'knowledge-document' as const, + id: item.id, + provenance: provenance.get(item.id) ?? { status: 'unknown' as const }, + view: { derived: item.value }, + })) + }, + writtenAs(options, 'knowledge-document') + ) + return new Set(admitted.map((item) => item.id)) +} + +/** + * Admits each chunk a result returns, on its own: its content's provenance merged with that of the + * document whose metadata it carries. `value` is what the result shows of the chunk; without one, + * every secret the chunk and its document can hold counts. Returns the ids admitted. + */ +export async function admitKnowledgeChunks( + principal: Principal, + chunks: readonly { + id: string + documentId: string + content: string + value?: unknown + snapshot?: KnowledgeDocumentSnapshot + }[], + options: KnowledgeAdmissionOptions = {} +): Promise> { + const admitted = await admitEach( + principal, + chunks, + async (items) => { + const provenance = await classifyKnowledgeChunks( + items.map((chunk) => ({ + ...chunk, + snapshot: chunk.snapshot ?? knowledgeDocumentSnapshot(chunk.value ?? chunk), + })) + ) + return items.map((chunk) => ({ + kind: 'knowledge-chunk' as const, + id: chunk.id, + provenance: provenance.get(chunk.id) ?? { status: 'unknown' as const }, + view: chunk.value === undefined ? ('complete' as const) : { derived: chunk.value }, + })) + }, + writtenAs(options, 'knowledge-chunk') + ) + return new Set(admitted.map((chunk) => chunk.id)) +} diff --git a/apps/sim/lib/knowledge/application/authorized-knowledge-use-case.ts b/apps/sim/lib/knowledge/application/authorized-knowledge-use-case.ts index ef5c7b98d1f..12a902ac374 100644 --- a/apps/sim/lib/knowledge/application/authorized-knowledge-use-case.ts +++ b/apps/sim/lib/knowledge/application/authorized-knowledge-use-case.ts @@ -5,6 +5,7 @@ import { type PrincipalForOperation, recordProjectedUseCaseAuditEntries, requireAllowedWorkspacePrincipal, + runDeclaredContentOperation, type WorkspaceUseCaseAuditEntry, } from '@/lib/core/application' import { authorizeOrganizationOperation } from '@/lib/core/application/organization-authorization' @@ -234,13 +235,18 @@ export function defineAuthorizedKnowledgeUseCase< context: resolved.context, request, } - return runWithOutboundOrganization(resolved.context.organizationId, async () => { - const result = await definition.execute(executionContext) - const resultContext = { ...executionContext, result } - recordOrganizationAudit(resultContext, resolved.context.organizationId) - await definition.afterSuccess?.(resultContext) - return result - }) + return runWithOutboundOrganization(resolved.context.organizationId, () => + runDeclaredContentOperation( + definition.operation, + resolved.principal, + () => definition.execute(executionContext), + async (result) => { + const resultContext = { ...executionContext, result } + recordOrganizationAudit(resultContext, resolved.context.organizationId) + await definition.afterSuccess?.(resultContext) + } + ) + ) }, } } diff --git a/apps/sim/lib/knowledge/application/chunks.test.ts b/apps/sim/lib/knowledge/application/chunks.test.ts index 5b7121d60f4..1475f02f69e 100644 --- a/apps/sim/lib/knowledge/application/chunks.test.ts +++ b/apps/sim/lib/knowledge/application/chunks.test.ts @@ -1,3 +1,4 @@ +import type { Principal } from '@sim/auth/principal' import { createSessionPrincipal } from '@sim/testing/factories/principal.factory' import { knowledgeContextsMock, @@ -8,6 +9,7 @@ import { workspaceAuthzMock, workspaceAuthzMockFns } from '@sim/testing/mocks/wo import { beforeEach, describe, expect, it, vi } from 'vitest' const mocks = vi.hoisted(() => ({ + admitKnowledgeChunks: vi.fn(), queryChunks: vi.fn(), batchChunkOperation: vi.fn(), createChunk: vi.fn(), @@ -36,10 +38,19 @@ vi.mock('@/lib/knowledge/model-input-provenance', () => ({ vi.mock('@/providers/utils', () => providersUtilsMock) +vi.mock('@/lib/knowledge/application/admit-knowledge-content', () => ({ + admitKnowledgeChunks: mocks.admitKnowledgeChunks, +})) + +import { admitContent } from '@/lib/core/application' import { ForbiddenOperationError } from '@/lib/core/application/forbidden' import { WORKSPACE_ACCESS_SCOPE } from '@/lib/knowledge/access/scope' import { KnowledgeDocumentNotReadyError } from '@/lib/knowledge/application/chunk-errors' -import { bulkUpdateKnowledgeChunks, listKnowledgeChunks } from '@/lib/knowledge/application/chunks' +import { + bulkUpdateKnowledgeChunks, + createKnowledgeChunk, + listKnowledgeChunks, +} from '@/lib/knowledge/application/chunks' describe('knowledge chunk application use cases', () => { beforeEach(() => { @@ -57,6 +68,45 @@ describe('knowledge chunk application use cases', () => { }) }) + /** Stands in for the gate's verdicts; accounting is the real primitive's, as the helper's is. */ + function admitOnly(ids: string[]) { + mocks.admitKnowledgeChunks.mockImplementation(async (principal: Principal) => { + await admitContent(principal, []) + return new Set(ids) + }) + } + + it('admits a created chunk with the document name its response shows', async () => { + workspaceAuthzMockFns.mockResolveEffectiveWorkspacePermission.mockResolvedValue('write') + knowledgeContextsMockFns.mockResolveCanonicalActiveKnowledgeDocumentContext.mockResolvedValue({ + workspaceId: 'workspace-1', + workspaceOrganizationId: null, + allowPersonalApiKeys: true, + billedAccountUserId: 'billing-owner-1', + access: { get: async () => WORKSPACE_ACCESS_SCOPE }, + knowledgeBaseId: 'knowledge-1', + knowledgeBase: { id: 'knowledge-1', embeddingModel: 'text-embedding-3-small' }, + documentId: 'document-1', + document: { id: 'document-1', processingStatus: 'completed', filename: 'named.txt' }, + }) + mocks.createChunk.mockResolvedValue({ id: 'chunk-new', content: 'body', tokenCount: 1 }) + admitOnly(['chunk-new']) + + const result = await createKnowledgeChunk.execute({ + principal: createSessionPrincipal(), + input: { + knowledgeBaseId: 'knowledge-1', + documentId: 'document-1', + content: 'body', + resolveContentProvenance: () => undefined, + }, + }) + + expect(result.chunk.documentName).toBe('named.txt') + const [, [item]] = mocks.admitKnowledgeChunks.mock.calls[0] + expect(item.value).toMatchObject({ documentId: 'document-1', documentName: 'named.txt' }) + }) + it('returns a typed transient failure before querying chunks for a processing document', async () => { const promise = listKnowledgeChunks.execute({ principal: createSessionPrincipal(), diff --git a/apps/sim/lib/knowledge/application/chunks.ts b/apps/sim/lib/knowledge/application/chunks.ts index fd2f5774014..719f87d56ed 100644 --- a/apps/sim/lib/knowledge/application/chunks.ts +++ b/apps/sim/lib/knowledge/application/chunks.ts @@ -1,5 +1,6 @@ import type { Principal } from '@sim/auth/principal' import { createLogger } from '@sim/logger' +import { ContentRefusedError } from '@/lib/core/application' import { ForbiddenOperationError } from '@/lib/core/application/forbidden' import { OrchestrationError } from '@/lib/core/orchestration/types' import { generateRequestId } from '@/lib/core/utils/request' @@ -8,6 +9,7 @@ import { type DurableSecretProvenance, } from '@/lib/execution/durable-secret-provenance' import { reportDurableSecretProvenanceRefusal } from '@/lib/execution/durable-secret-provenance-telemetry' +import { admitKnowledgeChunks } from '@/lib/knowledge/application/admit-knowledge-content' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { resolveKnowledgeAttributedUserId } from '@/lib/knowledge/application/billing' import { KnowledgeDocumentNotReadyError } from '@/lib/knowledge/application/chunk-errors' @@ -26,6 +28,7 @@ import { } from '@/lib/knowledge/chunks/service' import type { ChunkFilters } from '@/lib/knowledge/chunks/types' import { runWithKnowledgeModelInputProvenance } from '@/lib/knowledge/model-input-provenance' +import type { KnowledgeDocumentSnapshot } from '@/lib/knowledge/secret-provenance' import { calculateCost } from '@/providers/utils' const logger = createLogger('KnowledgeChunkApplication') @@ -65,6 +68,30 @@ export interface BulkKnowledgeChunksInput extends KnowledgeDocumentChunkInput { chunkIds: string[] } +/** + * One chunk a result returns. A read is refused on its own when its gate refuses it; a committed + * write completes and then answers with the refusal (see `admitWrittenContent`). + */ +async function requireAdmittedChunk( + principal: Principal, + documentId: string, + chunk: { id: string; content: string }, + options: { + written?: boolean + shown?: Record + snapshot?: KnowledgeDocumentSnapshot + } = {} +): Promise { + /** `shown` is every other field the response adds beside the chunk, such as its document name. */ + const value = options.shown ? { ...chunk, ...options.shown } : chunk + const admitted = await admitKnowledgeChunks( + principal, + [{ id: chunk.id, documentId, content: chunk.content, value, snapshot: options.snapshot }], + { written: options.written } + ) + if (!options.written && !admitted.has(chunk.id)) throw new ContentRefusedError('knowledge-chunk') +} + function requireChunkReadable(context: ActiveKnowledgeDocumentContext): void { if (context.document.processingStatus !== 'completed') { throw new KnowledgeDocumentNotReadyError(context.document.processingStatus) @@ -112,7 +139,7 @@ export const listKnowledgeChunks = defineAuthorizedKnowledgeUseCase({ principal: Principal input: ListKnowledgeChunksInput }) => resolveCanonicalActiveKnowledgeDocumentContext(input, principal), - async execute({ input, context }) { + async execute({ principal, input, context }) { if (input.requireEnabledDocument && !context.document.enabled) { throw new OrchestrationError('not_found', 'Document not found') } @@ -130,7 +157,22 @@ export const listKnowledgeChunks = defineAuthorizedKnowledgeUseCase({ generateRequestId(), await context.access.get() ) - return { ...result, workspaceId: context.workspaceId, documentId } + const admitted = await admitKnowledgeChunks( + principal, + result.chunks.map((chunk) => ({ + id: chunk.id, + documentId, + content: chunk.content, + value: chunk, + })) + ) + const chunks = result.chunks.filter((chunk) => admitted.has(chunk.id)) + return { + ...result, + chunks, + workspaceId: context.workspaceId, + documentId, + } }, }) @@ -138,8 +180,9 @@ export const readKnowledgeChunk = defineAuthorizedKnowledgeUseCase({ operation: knowledgeOperations.readChunk, resolveContext: ({ principal, input }: { principal: Principal; input: KnowledgeChunkInput }) => resolveActiveKnowledgeChunkContext(input, principal), - async execute({ context }) { + async execute({ principal, context }) { requireChunkReadable(context) + await requireAdmittedChunk(principal, context.documentId, context.chunk) return { chunk: context.chunk, workspaceId: context.workspaceId, @@ -196,6 +239,11 @@ export const createKnowledgeChunk = defineAuthorizedKnowledgeUseCase({ } catch (error) { logger.warn('Failed to calculate cost for chunk upload', { error }) } + await requireAdmittedChunk(principal, context.documentId, chunk, { + written: true, + shown: { documentId: context.documentId, documentName: context.document.filename }, + snapshot: { filename: context.document.filename }, + }) return { chunk: { ...chunk, @@ -259,6 +307,7 @@ export const updateKnowledgeChunk = defineAuthorizedKnowledgeUseCase({ provenance ) ) + await requireAdmittedChunk(principal, context.documentId, chunk, { written: true }) return { chunk, workspaceId: context.workspaceId, documentId: context.documentId } }, }) diff --git a/apps/sim/lib/knowledge/application/connectors.ts b/apps/sim/lib/knowledge/application/connectors.ts index aee24d4ba78..b60670c0b40 100644 --- a/apps/sim/lib/knowledge/application/connectors.ts +++ b/apps/sim/lib/knowledge/application/connectors.ts @@ -35,6 +35,7 @@ import { parseExactEnvironmentReference } from '@/lib/environment/reference' import { resolveEffectiveEnvironmentVariables } from '@/lib/environment/utils' import { requireKnowledgeMemberAccessAvailable } from '@/lib/knowledge/access/availability' import { knowledgeAccessCondition } from '@/lib/knowledge/access/predicate' +import { admitKnowledgeDocuments } from '@/lib/knowledge/application/admit-knowledge-content' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { resolveKnowledgeAttributedUserId, @@ -1190,7 +1191,7 @@ export const listKnowledgeConnectorDocuments = defineAuthorizedKnowledgeUseCase( principal: Principal input: ListKnowledgeConnectorDocumentsInput }) => resolveActiveKnowledgeConnectorContext(input, principal), - async execute({ input, context }) { + async execute({ principal, input, context }) { const limit = input.limit ?? DEFAULT_KNOWLEDGE_CONNECTOR_DOCUMENT_PAGE_SIZE const offset = input.offset ?? 0 if ( @@ -1218,15 +1219,16 @@ export const listKnowledgeConnectorDocuments = defineAuthorizedKnowledgeUseCase( } const filter = input.filter ?? (input.failedOnly ? 'failed' : input.includeExcluded ? undefined : 'active') - const baseConditions = [ + const scopeConditions = [ eq(document.connectorId, context.connectorId), isNull(document.archivedAt), isNull(document.deletedAt), knowledgeAccessCondition(await context.access.get()), - search - ? sql`${document.filename} ILIKE ${`%${escapeLikePattern(search)}%`} ESCAPE '\\'` - : undefined, ] as const + const searchCondition = search + ? sql`${document.filename} ILIKE ${`%${escapeLikePattern(search)}%`} ESCAPE '\\'` + : undefined + const baseConditions = [...scopeConditions, searchCondition] as const const active = sql`${document.userExcluded} = false` /** One pass over the source's readable documents; excluded rows are scanned only when counted. */ const [[counts], rows] = await Promise.all([ @@ -1260,9 +1262,13 @@ export const listKnowledgeConnectorDocuments = defineAuthorizedKnowledgeUseCase( .offset(offset), ]) const hasMore = rows.length > limit - const documents = rows.slice(0, limit) + const page = rows.slice(0, limit) + const admitted = await admitKnowledgeDocuments( + principal, + page.map((row) => ({ id: row.id, value: row })) + ) return { - documents, + documents: page.filter((row) => admitted.has(row.id)), counts: { active: counts?.active ?? 0, excluded: counts?.excluded ?? 0, diff --git a/apps/sim/lib/knowledge/application/documents.test.ts b/apps/sim/lib/knowledge/application/documents.test.ts index 31ceaca2b64..25761ec99b6 100644 --- a/apps/sim/lib/knowledge/application/documents.test.ts +++ b/apps/sim/lib/knowledge/application/documents.test.ts @@ -68,6 +68,27 @@ vi.mock('@/lib/uploads/server/metadata', () => uploadsMetadataMock) vi.mock('@/lib/posthog/server', () => posthogServerMock) +/** Chat's verdicts come from a stand-in gate (refusing `withheld-*` ids); other callers run the real helper. */ +vi.mock('@/lib/knowledge/application/admit-knowledge-content', async (importOriginal) => { + const original = + await importOriginal() + const delivery = + await vi.importActual('@/lib/core/application') + return { + ...original, + admitKnowledgeDocuments: async ( + principal: Parameters[0], + documents: Parameters[1] + ) => { + if (!delivery.isCopilotPrincipal(principal)) { + return original.admitKnowledgeDocuments(principal, documents) + } + await delivery.admitContent(principal, []) + return new Set(documents.map((doc) => doc.id).filter((id) => !id.startsWith('withheld-'))) + }, + } +}) + import { OrchestrationError } from '@/lib/core/orchestration/types' import { WORKSPACE_ACCESS_SCOPE } from '@/lib/knowledge/access/scope' import { diff --git a/apps/sim/lib/knowledge/application/documents.ts b/apps/sim/lib/knowledge/application/documents.ts index 6a29c50085f..300ecfb3640 100644 --- a/apps/sim/lib/knowledge/application/documents.ts +++ b/apps/sim/lib/knowledge/application/documents.ts @@ -8,9 +8,14 @@ import { type BillingAttributionSnapshot, checkAttributedUsageLimits, } from '@/lib/billing/core/billing-attribution' -import { authorizeWorkspaceOperation } from '@/lib/core/application' +import { + admitContent, + authorizeWorkspaceOperation, + ContentRefusedError, +} from '@/lib/core/application' import { OrchestrationError } from '@/lib/core/orchestration/types' import { generateRequestId } from '@/lib/core/utils/request' +import { admitKnowledgeDocuments } from '@/lib/knowledge/application/admit-knowledge-content' import { knowledgeDelegationPolicy } from '@/lib/knowledge/application/authorization' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { @@ -257,6 +262,25 @@ export interface UpsertKnowledgeDocumentInput extends UploadKnowledgeDocumentAdm * so a presenter can key each document's tag values by display name — the same * projection knowledge search performs — without reading protected data itself. */ +/** + * One document a result returns. A read is refused on its own when its gate refuses its metadata; + * a committed write completes and then answers with the refusal (see `admitWrittenContent`). + */ +async function requireAdmittedDocument( + principal: Principal, + document: { id: string }, + options: { written?: boolean } = {} +): Promise { + const admitted = await admitKnowledgeDocuments( + principal, + [{ id: document.id, value: document }], + options + ) + if (!options.written && !admitted.has(document.id)) { + throw new ContentRefusedError('knowledge-document') + } +} + export const listKnowledgeDocuments = defineAuthorizedKnowledgeUseCase({ operation: knowledgeOperations.listDocuments, resolveContext: ({ @@ -266,7 +290,7 @@ export const listKnowledgeDocuments = defineAuthorizedKnowledgeUseCase({ principal: Principal input: ListKnowledgeDocumentsInput }) => resolveActiveKnowledgeResourceContext(input, principal), - async execute({ input, context }) { + async execute({ principal, input, context }) { const limit = input.limit ?? 50 const offset = input.offset ?? 0 if (!Number.isInteger(limit) || limit < 1 || limit > 100) { @@ -298,8 +322,14 @@ export const listKnowledgeDocuments = defineAuthorizedKnowledgeUseCase({ generateRequestId(), context.access ) + const admitted = await admitKnowledgeDocuments( + principal, + result.documents.map((doc) => ({ id: doc.id, value: doc })) + ) + const documents = result.documents.filter((doc) => admitted.has(doc.id)) return { ...result, + documents, tagDefinitions: resolvedNameFilters?.definitionsByKnowledgeBase.get(context.knowledgeBaseId) ?? (await getDocumentTagDefinitions(context.knowledgeBaseId)), @@ -317,10 +347,11 @@ export const readKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ principal: Principal input: ReadKnowledgeDocumentInput }) => resolveActiveKnowledgeDocumentContext(input, principal), - async execute({ input, context }) { + async execute({ principal, input, context }) { if (input.requireEnabledDocument && !context.document.enabled) { throw new OrchestrationError('not_found', 'Document not found') } + await requireAdmittedDocument(principal, context.document) return { document: context.document, tagDefinitions: await getDocumentTagDefinitions(context.knowledgeBaseId), @@ -828,6 +859,7 @@ export const updateKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ } throw new OrchestrationError(outcome.errorCode, outcome.error) } + await admitContent(principal, []) return { kind: 'processing' as const, documentId: context.documentId, @@ -853,9 +885,11 @@ export const updateKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ if (updatedFields.length === 0) { throw new OrchestrationError('validation', 'No updates specified') } + const document = await updateDocument(context.documentId, updates, generateRequestId()) + await requireAdmittedDocument(principal, document, { written: true }) return { kind: 'updated' as const, - document: await updateDocument(context.documentId, updates, generateRequestId()), + document, tagDefinitions: await getDocumentTagDefinitions(context.knowledgeBaseId), updatedFields, } diff --git a/apps/sim/lib/knowledge/application/exports.ts b/apps/sim/lib/knowledge/application/exports.ts index 646374f806c..38bb7054702 100644 --- a/apps/sim/lib/knowledge/application/exports.ts +++ b/apps/sim/lib/knowledge/application/exports.ts @@ -1,10 +1,23 @@ +import { AsyncLocalStorage } from 'node:async_hooks' +import type { Readable } from 'node:stream' import { AuditAction, AuditResourceType } from '@sim/audit' import type { Principal } from '@sim/auth/principal' +import { + admitContent, + ContentRefusedError, + mustClassifyContent, + requireAdmittedContent, +} from '@/lib/core/application' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { resolveActiveKnowledgeBaseContext } from '@/lib/knowledge/application/contexts' import { knowledgeOperations } from '@/lib/knowledge/application/operations' import { KNOWLEDGE_BUNDLE_VERSION } from '@/lib/knowledge/constants' import { toKbEmbeddingDimensions } from '@/lib/knowledge/embedding-models' +import { + classifyKnowledgeChunkContent, + classifyKnowledgeDocumentMetadata, + loadKnowledgeDocumentDurableSecretProvenance, +} from '@/lib/knowledge/secret-provenance' import { bundleEntryPaths, type KnowledgeBundleManifest, @@ -12,6 +25,7 @@ import { parseDescribableBundle, toManifestDocument, } from '@/lib/knowledge/transfer/bundle' +import { openKnowledgeExportFile } from '@/lib/knowledge/transfer/export-archive' import { type ExportableChunk, type ExportableDocument, @@ -38,6 +52,7 @@ export interface KnowledgeBaseExportBundle { embedding: KnowledgeBundleManifest['embedding'] tags: KnowledgeBundleTag[] documents: ExportableDocument[] + file(document: ExportableDocument): Promise chunks(documentId: string): AsyncIterable } @@ -50,13 +65,22 @@ export const exportKnowledgeBase = defineAuthorizedKnowledgeUseCase({ principal: Principal input: ExportKnowledgeBaseInput }) => resolveActiveKnowledgeBaseContext(input, principal), - async execute({ input, context }): Promise { + async execute({ principal, input, context }): Promise { const { knowledgeBase } = context const dimension = toKbEmbeddingDimensions(knowledgeBase.embeddingDimension) const [tags, documents] = await Promise.all([ listExportableTags(knowledgeBase.id), listExportableDocuments(knowledgeBase.id), ]) + const classify = mustClassifyContent(principal) + const inExportContext = AsyncLocalStorage.snapshot() + const documentById = new Map(documents.map((document) => [document.id, document])) + const snapshot = (document: ExportableDocument) => ({ + filename: document.filename, + ...document.tags, + }) + // Deferred bytes are admitted in the captured context as their stream reaches each source. + await admitContent(principal, []) const manifest = parseDescribableBundle({ version: KNOWLEDGE_BUNDLE_VERSION, exportedAt: new Date().toISOString(), @@ -80,8 +104,69 @@ export const exportKnowledgeBase = defineAuthorizedKnowledgeUseCase({ embedding: manifest.embedding, tags: manifest.tags, documents, - chunks: (documentId) => - iterateDocumentChunks(knowledgeBase.id, documentId, input.vectors ? dimension : null), + file: (document) => + inExportContext(async () => { + if (!document.file) throw new ContentRefusedError('knowledge-document') + if (!classify) return openKnowledgeExportFile(document.file) + const current = await loadKnowledgeDocumentDurableSecretProvenance( + document.id, + snapshot(document) + ) + if ( + current.knowledgeBaseId !== knowledgeBase.id || + (document.file.kind === 'storage' && current.storageKey !== document.file.key) || + (document.file.kind === 'data-uri' && current.storageKey !== null) + ) { + throw new ContentRefusedError('knowledge-document') + } + await requireAdmittedContent(principal, { + kind: 'knowledge-document', + id: document.id, + provenance: current.provenance, + view: 'opaque', + }) + return openKnowledgeExportFile(document.file, current.source.fileUrl) + }), + chunks: async function* (documentId) { + const document = documentById.get(documentId) + if (!document) throw new ContentRefusedError('knowledge-document') + if (classify) { + await inExportContext(async () => { + const provenance = await classifyKnowledgeDocumentMetadata( + [documentId], + new Map([[documentId, [snapshot(document)]]]) + ) + await requireAdmittedContent(principal, { + kind: 'knowledge-document', + id: documentId, + provenance: provenance.get(documentId) ?? { status: 'unknown' }, + view: 'opaque', + }) + }) + } + yield* iterateDocumentChunks( + knowledgeBase.id, + documentId, + input.vectors ? dimension : null, + classify + ? (page) => + inExportContext(async () => { + const provenance = await classifyKnowledgeChunkContent( + page.map((chunk) => ({ ...chunk, documentId })) + ) + const sources = page.map((chunk) => ({ + kind: 'knowledge-chunk' as const, + id: chunk.id, + provenance: provenance.get(chunk.id) ?? { status: 'unknown' as const }, + view: 'opaque' as const, + })) + if ((await admitContent(principal, sources)).refused.length > 0) { + throw new ContentRefusedError('knowledge-chunk') + } + }) + : undefined + ) + }, } }, projectAudit: ({ context, input, result }) => ({ diff --git a/apps/sim/lib/knowledge/application/operations.ts b/apps/sim/lib/knowledge/application/operations.ts index 28202ce3788..b3f3cc58ad6 100644 --- a/apps/sim/lib/knowledge/application/operations.ts +++ b/apps/sim/lib/knowledge/application/operations.ts @@ -66,6 +66,8 @@ function defineKnowledgeOperation( delegationAudience: options?.organizationDelegationAudience ?? 'sim:knowledge', delegatedServices: operation.id === 'knowledge.search' ? ['copilot', 'slack-search'] : ['copilot'], + /** Chat reaches the organization form through the same use case, under one declaration. */ + contentDelivery: operation.contentDelivery ?? 'none', } as const) : ({ principalKinds: ['session', 'personal_api_key', 'oauth_access_token'] } as const)), }) @@ -275,6 +277,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -285,6 +288,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -300,6 +304,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.export', + contentDelivery: 'sourced', ...ALL_PRINCIPAL_POLICY, }) ), @@ -315,6 +320,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.create', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -325,6 +331,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -335,6 +342,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -352,6 +360,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -362,6 +371,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -372,6 +382,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -382,6 +393,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -392,6 +404,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'sourced', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -402,6 +415,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), /** Folder mentions resolve workspace folders; organization delegation stays disabled. */ @@ -414,6 +428,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -424,6 +439,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -434,6 +450,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -444,6 +461,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'sourced', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -454,6 +472,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'sourced', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -470,6 +489,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.upload', + contentDelivery: 'none', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -480,6 +500,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -490,6 +511,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -500,6 +522,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -510,6 +533,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -520,6 +544,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'sourced', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -530,6 +555,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -540,6 +566,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -550,6 +577,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -560,6 +588,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -570,6 +599,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -586,6 +616,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_WITH_EXECUTOR_POLICY, }) ), @@ -596,6 +627,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -606,6 +638,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -616,6 +649,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -626,6 +660,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -636,6 +671,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -646,6 +682,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -663,6 +700,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -674,6 +712,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -684,6 +723,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -694,6 +734,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -704,6 +745,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -714,6 +756,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -746,6 +789,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }) @@ -756,6 +800,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', principalKinds: ['session', 'delegated'], delegatedServices: ['copilot'], }) @@ -800,6 +845,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'knowledge.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -820,6 +866,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -830,6 +877,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_COPILOT_AND_EXECUTOR_PRINCIPAL_POLICY, }) ), @@ -840,6 +888,7 @@ export const knowledgeOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'sourced', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -850,6 +899,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'deny', capability: 'knowledge.use', + contentDelivery: 'none', ...HUMAN_AND_COPILOT_PRINCIPAL_POLICY, }) ), @@ -866,6 +916,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.upload', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -876,6 +927,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.upload', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -886,6 +938,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.upload', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), @@ -896,6 +949,7 @@ export const knowledgeOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'knowledge.upload', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }) ), diff --git a/apps/sim/lib/knowledge/application/read-original-document.test.ts b/apps/sim/lib/knowledge/application/read-original-document.test.ts index 4ab20423c08..9df9cc92ba4 100644 --- a/apps/sim/lib/knowledge/application/read-original-document.test.ts +++ b/apps/sim/lib/knowledge/application/read-original-document.test.ts @@ -21,8 +21,15 @@ vi.mock('@/lib/knowledge/secret-provenance', () => ({ loadKnowledgeDocumentDurableSecretProvenance: mocks.provenance, })) +import { + type ContentDeliveryGate, + ContentRefusedError, + observeContentDelivery, +} from '@/lib/core/application' import { OrchestrationError } from '@/lib/core/orchestration/types' +import { KNOWLEDGE_DELEGATION_AUDIENCE } from '@/lib/knowledge/application/authorization' import { readOriginalKnowledgeDocument } from '@/lib/knowledge/application/read-original-document' +import { createCopilotChatPrincipal } from '@/lib/mothership/auth/application-delegation' workspaceAuthzMockFns.mockPermissionSatisfies.mockImplementation( (actual: string | null) => actual !== null @@ -54,6 +61,27 @@ const input = { } const run = () => readOriginalKnowledgeDocument.execute({ principal, input }) +/** A gate with no registry: it admits only items with no recorded secret. */ +const secretFreeGate: ContentDeliveryGate = { + async admit(sources) { + return sources.map( + ({ provenance }) => + provenance.status === 'unrecorded' || + (provenance.status === 'exact' && provenance.entries.length === 0) + ) + }, +} +const runAsChat = () => + observeContentDelivery(secretFreeGate, () => + readOriginalKnowledgeDocument.execute({ + principal: createCopilotChatPrincipal( + { userId: 'reader', workspaceId: 'workspace', chatId: 'chat' }, + KNOWLEDGE_DELEGATION_AUDIENCE + ), + input, + }) + ) + describe('original knowledge document read', () => { beforeEach(() => { knowledgeContextsMockFns.mockResolveCanonicalActiveKnowledgeDocumentContext.mockResolvedValue( @@ -61,6 +89,7 @@ describe('original knowledge document read', () => { ) workspaceAuthzMockFns.mockResolveEffectiveWorkspacePermission.mockResolvedValue('read') mocks.provenance.mockResolvedValue({ + storageKey: document.storageKey, source: { filename: document.filename, fileUrl: document.fileUrl }, provenance: { status: 'exact', entries: [] }, }) @@ -76,6 +105,7 @@ describe('original knowledge document read', () => { }, }) mocks.provenance.mockResolvedValue({ + storageKey: document.storageKey, source: { filename: document.filename, fileUrl: 'https://example.test/private' }, provenance: { status: 'exact', entries: [] }, }) @@ -113,21 +143,35 @@ describe('original knowledge document read', () => { ).rejects.toThrow(/limit/) }) it.each([{ status: 'unknown' }, { status: 'exact', entries: [{ secret: 'private' }] }])( - 'refuses originals without secret-free provenance', + 'refuses Chat an original its gate refuses, before reading it: %j', async (provenance) => { mocks.provenance.mockResolvedValue({ + storageKey: document.storageKey, source: { filename: document.filename, fileUrl: document.fileUrl }, provenance, }) - await expect(run()).rejects.toThrow('secret-free') + await expect(runAsChat()).rejects.toBeInstanceOf(ContentRefusedError) expect(storageServiceMockFns.mockDownloadFile).not.toHaveBeenCalled() } ) - it('refuses a concurrent source replacement before reading', async () => { + it('serves Chat a secret-free original under its gate', async () => { + expect(await runAsChat()).toMatchObject({ sourceAvailable: true, hasRecordedSecrets: false }) + }) + it('classifies a concurrent source replacement as unknown, refusing it before reading', async () => { mocks.provenance.mockResolvedValue({ + storageKey: document.storageKey, source: { filename: 'changed', fileUrl: document.fileUrl }, provenance: { status: 'exact', entries: [] }, }) - await expect(run()).rejects.toThrow('secret-free') + await expect(runAsChat()).rejects.toBeInstanceOf(ContentRefusedError) + expect(storageServiceMockFns.mockDownloadFile).not.toHaveBeenCalled() + }) + it('reads for a caller outside Chat without classifying, flagging recorded secrets', async () => { + mocks.provenance.mockResolvedValue({ + storageKey: document.storageKey, + source: { filename: document.filename, fileUrl: document.fileUrl }, + provenance: { status: 'exact', entries: [{ secret: 'private' }] }, + }) + expect(await run()).toMatchObject({ sourceAvailable: true, hasRecordedSecrets: true }) }) }) diff --git a/apps/sim/lib/knowledge/application/read-original-document.ts b/apps/sim/lib/knowledge/application/read-original-document.ts index 2af16362da2..40c99c5038b 100644 --- a/apps/sim/lib/knowledge/application/read-original-document.ts +++ b/apps/sim/lib/knowledge/application/read-original-document.ts @@ -1,4 +1,5 @@ import type { Principal } from '@sim/auth/principal' +import { requireAdmittedContent } from '@/lib/core/application' import { OrchestrationError } from '@/lib/core/orchestration/types' import { hashDurableSecretProvenanceValue } from '@/lib/execution/durable-secret-provenance' import { decodeDataUriWithinLimit } from '@/lib/file-parsers/data-uri' @@ -31,7 +32,7 @@ export const readOriginalKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ principal: Principal input: ReadOriginalKnowledgeDocumentInput }) => resolveCanonicalActiveKnowledgeDocumentContext(input, principal), - async execute({ input, context }) { + async execute({ input, context, principal }) { const maxBytes = input.maxBytes ?? MAX_TEXT_EXTRACTION_BYTES if (!Number.isSafeInteger(maxBytes) || maxBytes < 1 || maxBytes > MAX_TEXT_EXTRACTION_BYTES) throw new OrchestrationError('validation', 'Invalid document read byte limit') @@ -45,23 +46,31 @@ export const readOriginalKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ processingStatus: doc.processingStatus, indexReady: doc.enabled && doc.processingStatus === 'completed', } - const unavailable = { ...metadata, sourceAvailable: false as const, buffer: undefined } if (doc.fileSize > maxBytes) throw new OrchestrationError( 'payload_too_large', 'Document exceeds the source read byte limit' ) const snapshot = await loadKnowledgeDocumentDurableSecretProvenance(doc.id) - if ( - hashDurableSecretProvenanceValue(snapshot.source) !== - hashDurableSecretProvenanceValue(createKnowledgeDocumentSourceValue(doc)) || - snapshot.provenance.status !== 'exact' || - snapshot.provenance.entries.length !== 0 - ) { - throw new OrchestrationError( - 'forbidden', - 'This document has no verified secret-free original. Read its projected indexed passages instead.' - ) + const provenance = + snapshot.storageKey === doc.storageKey && + hashDurableSecretProvenanceValue(snapshot.source) === + hashDurableSecretProvenanceValue(createKnowledgeDocumentSourceValue(doc)) + ? snapshot.provenance + : ({ status: 'unknown' } as const) + await requireAdmittedContent(principal, { + kind: 'knowledge-document', + id: doc.id, + provenance, + view: 'complete', + }) + /** A visual of these bytes would show what text redaction hides. */ + const hasRecordedSecrets = provenance.status === 'exact' && provenance.entries.length > 0 + const unavailable = { + ...metadata, + sourceAvailable: false as const, + buffer: undefined, + hasRecordedSecrets, } if (!doc.storageKey && !doc.fileUrl.startsWith('data:')) return unavailable let buffer: Buffer @@ -84,6 +93,6 @@ export const readOriginalKnowledgeDocument = defineAuthorizedKnowledgeUseCase({ throw error } input.signal?.throwIfAborted() - return { ...metadata, sourceAvailable: true as const, buffer } + return { ...metadata, sourceAvailable: true as const, buffer, hasRecordedSecrets } }, }) diff --git a/apps/sim/lib/knowledge/application/search.ts b/apps/sim/lib/knowledge/application/search.ts index 515b3f73826..40b02571c8f 100644 --- a/apps/sim/lib/knowledge/application/search.ts +++ b/apps/sim/lib/knowledge/application/search.ts @@ -8,6 +8,7 @@ import { import { checkSearchUsageLimits } from '@/lib/billing/core/usage-gate-cache' import { recordUsage } from '@/lib/billing/core/usage-log' import { checkAndBillPayerOverageThreshold } from '@/lib/billing/threshold-billing' +import { admitContent } from '@/lib/core/application' import { OrchestrationError } from '@/lib/core/orchestration/types' import { resourceScopeFromOwner, resourceScopeKey } from '@/lib/core/resource-scope' import { PlatformEvents } from '@/lib/core/telemetry' @@ -17,6 +18,7 @@ import { reportDurableSecretProvenanceRefusal } from '@/lib/execution/durable-se import { requireOrganizationSearchAvailable } from '@/lib/knowledge/access/availability' import { createKnowledgeAccessProvider } from '@/lib/knowledge/access/scope' import type { KnowledgeAccessProvider } from '@/lib/knowledge/access/types' +import { admitKnowledgeChunks } from '@/lib/knowledge/application/admit-knowledge-content' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { KnowledgeUsageLimitExceededError, @@ -156,7 +158,7 @@ interface KnowledgeSearchCost { rerankerSearchUnits?: number } -export interface SearchKnowledgeResult { +interface SearchKnowledgeResult { retrieval: RetrievalStatus results: KnowledgeSearchItem[] query: string @@ -480,7 +482,14 @@ export async function runKnowledgeSearch({ }) if (retrieved.retrieval.status === 'partial' && !input.allowPartialResults) throw new SearchDeadlineError() - let rows = retrieved.rows + input.signal?.throwIfAborted() + /** + * Chat sees each chunk only as its own gate admits it, before the reranker or the result: one + * chunk without provenance is left out instead of withholding the search. A result also renders + * its document's name and tags, so every secret the chunk and its document hold counts. + */ + const admittedRows = await admitKnowledgeChunks(principal, retrieved.rows) + let rows = retrieved.rows.filter((row) => admittedRows.has(row.id)) input.signal?.throwIfAborted() /** Public callers have no input envelope, but persisted reranker inputs still need provenance. */ const registrySubjectUserId = resolvePrincipalSubjectUserId(principal) @@ -806,7 +815,11 @@ const searchKnowledgeUseCase = defineAuthorizedKnowledgeUseCase({ operation: knowledgeOperations.search, resolveContext: ({ principal, input }: { principal: Principal; input: SearchKnowledgeInput }) => measureSearchStage('knowledge_context', () => resolveKnowledgeSearchContext(input, principal)), - execute: ({ principal, input, context }) => runKnowledgeSearch({ principal, input, context }), + async execute({ principal, input, context }) { + /** Accounted up front: a search that retrieves nothing returns no chunk to admit. */ + await admitContent(principal, []) + return runKnowledgeSearch({ principal, input, context }) + }, afterSuccess: ({ principal, context, input, result }) => afterKnowledgeSearch({ principal, context, input, result }), }) diff --git a/apps/sim/lib/knowledge/application/tags.test.ts b/apps/sim/lib/knowledge/application/tags.test.ts index 8cf227c05f7..59e17fe3316 100644 --- a/apps/sim/lib/knowledge/application/tags.test.ts +++ b/apps/sim/lib/knowledge/application/tags.test.ts @@ -1,4 +1,8 @@ -import { createSessionPrincipal } from '@sim/testing/factories/principal.factory' +import type { Principal } from '@sim/auth/principal' +import { + createDelegatedPrincipal, + createSessionPrincipal, +} from '@sim/testing/factories/principal.factory' import { auditMock, auditMockFns } from '@sim/testing/mocks/audit.mock' import { knowledgeContextsMock, @@ -19,11 +23,16 @@ vi.mock('@/lib/knowledge/application/contexts', () => knowledgeContextsMock) vi.mock('@/lib/knowledge/tags/service', () => knowledgeTagsServiceMock) +const admission = vi.hoisted(() => ({ admitKnowledgeDocuments: vi.fn() })) +vi.mock('@/lib/knowledge/application/admit-knowledge-content', () => admission) + +import { admitContent } from '@/lib/core/application' import { WORKSPACE_ACCESS_SCOPE } from '@/lib/knowledge/access/scope' import { createKnowledgeTag, deleteKnowledgeTag, listKnowledgeTags, + readDetailedKnowledgeTagUsage, readKnowledgeTagUsage, readNextKnowledgeTagSlot, updateKnowledgeTag, @@ -74,17 +83,13 @@ const documentContext = { const sessionPrincipal = createSessionPrincipal() -const delegatedPrincipal = { - kind: 'delegated' as const, - serviceId: 'copilot', +const delegatedPrincipal = createDelegatedPrincipal({ subjectUserId: 'shared-user', workspaceId: 'workspace-a', delegationId: 'tool-call-1', audience: 'sim:knowledge', - issuedAt: new Date(), - expiresAt: new Date(Date.now() + 60_000), resourceScope: {}, -} +}) const mocks = { listTags: knowledgeTagsServiceMockFns.mockGetDocumentTagDefinitions, @@ -117,63 +122,73 @@ describe('knowledge tag application use cases', () => { }) it.each([ - [ - 'list', - listKnowledgeTags, - { knowledgeBaseId: 'knowledge-b', assertedWorkspaceId: 'workspace-a' }, - ], - [ - 'create', - createKnowledgeTag, - { - knowledgeBaseId: 'knowledge-b', - assertedWorkspaceId: 'workspace-a', - displayName: 'Region', - }, - ], - [ - 'update', - updateKnowledgeTag, - { - tagDefinitionId: 'tag-b', - assertedWorkspaceId: 'workspace-a', - updates: { displayName: 'Market' }, - }, - ], - [ - 'delete', - deleteKnowledgeTag, - { - knowledgeBaseId: 'knowledge-b', - tagDefinitionId: 'tag-b', - assertedWorkspaceId: 'workspace-a', - }, - ], - [ - 'read usage', - readKnowledgeTagUsage, - { knowledgeBaseId: 'knowledge-b', assertedWorkspaceId: 'workspace-a' }, - ], - ])( - 'rejects cross-workspace %s before current membership or tag work', - async (_name, useCase, input) => { - await expect(useCase.execute({ principal: delegatedPrincipal, input })).rejects.toMatchObject( - { - name: 'DelegatedWorkspaceAuthorizationError', - code: 'forbidden', - } - ) + { + name: 'list', + run: () => + listKnowledgeTags.execute({ + principal: delegatedPrincipal, + input: { knowledgeBaseId: 'knowledge-b', assertedWorkspaceId: 'workspace-a' }, + }), + }, + { + name: 'create', + run: () => + createKnowledgeTag.execute({ + principal: delegatedPrincipal, + input: { + knowledgeBaseId: 'knowledge-b', + assertedWorkspaceId: 'workspace-a', + displayName: 'Region', + }, + }), + }, + { + name: 'update', + run: () => + updateKnowledgeTag.execute({ + principal: delegatedPrincipal, + input: { + tagDefinitionId: 'tag-b', + assertedWorkspaceId: 'workspace-a', + updates: { displayName: 'Market' }, + }, + }), + }, + { + name: 'delete', + run: () => + deleteKnowledgeTag.execute({ + principal: delegatedPrincipal, + input: { + knowledgeBaseId: 'knowledge-b', + tagDefinitionId: 'tag-b', + assertedWorkspaceId: 'workspace-a', + }, + }), + }, + { + name: 'read usage', + run: () => + readKnowledgeTagUsage.execute({ + principal: delegatedPrincipal, + input: { knowledgeBaseId: 'knowledge-b', assertedWorkspaceId: 'workspace-a' }, + }), + }, + ])('rejects cross-workspace $name before current membership or tag work', async ({ run }) => { + await expect(run()).rejects.toMatchObject({ + name: 'DelegatedWorkspaceAuthorizationError', + code: 'forbidden', + }) - expect(mocks.resolvePermission).not.toHaveBeenCalled() - expect(mocks.listTags).not.toHaveBeenCalled() - expect(mocks.nextSlot).not.toHaveBeenCalled() - expect(mocks.createTag).not.toHaveBeenCalled() - expect(mocks.updateTag).not.toHaveBeenCalled() - expect(mocks.deleteTag).not.toHaveBeenCalled() - expect(mocks.readUsage).not.toHaveBeenCalled() - expect(mocks.recordAudit).not.toHaveBeenCalled() - } - ) + expect(mocks.resolvePermission).not.toHaveBeenCalled() + expect(mocks.listTags).not.toHaveBeenCalled() + expect(mocks.nextSlot).not.toHaveBeenCalled() + expect(mocks.createTag).not.toHaveBeenCalled() + expect(mocks.updateTag).not.toHaveBeenCalled() + expect(mocks.deleteTag).not.toHaveBeenCalled() + expect(mocks.readUsage).not.toHaveBeenCalled() + expect(mocks.recordAudit).not.toHaveBeenCalled() + }) it.each([ ['an unknown slot', { tagSlot: 'tag99', fieldType: 'text' }], @@ -353,4 +368,100 @@ describe('knowledge tag application use cases', () => { }) } ) + + describe('tag usage Chat may see', () => { + const chat = { ...delegatedPrincipal, workspaceId: 'workspace-b' } + /** One document under two definitions: its value is refused under one and admitted under the other. */ + const detailed = [ + { + tagName: 'Region', + tagSlot: 'tag1', + documentCount: 2, + documents: [ + { id: 'doc-shared', name: 'shared.csv', tagValue: 'refused' }, + { id: 'doc-other', name: 'other.csv', tagValue: 'emea' }, + ], + }, + { + tagName: 'Owner', + tagSlot: 'tag2', + documentCount: 1, + documents: [{ id: 'doc-shared', name: 'shared.csv', tagValue: 'ops' }], + }, + ] + + beforeEach(() => { + mocks.resolvePermission.mockResolvedValue('read') + mocks.readDetailedUsage.mockResolvedValue(detailed) + /** Stands in for the gate: refuses an occurrence by the value it shows. */ + admission.admitKnowledgeDocuments.mockImplementation( + async (principal: Principal, documents: { id: string; value: { tagValue: string } }[]) => { + await admitContent(principal, []) + return new Set( + documents.filter((doc) => doc.value.tagValue !== 'refused').map((doc) => doc.id) + ) + } + ) + }) + + it('answers Chat an empty usage for a base with no tag definitions', async () => { + mocks.readDetailedUsage.mockResolvedValue([]) + await expect( + readDetailedKnowledgeTagUsage.execute({ + principal: chat, + input: { knowledgeBaseId: 'knowledge-b' }, + }) + ).resolves.toMatchObject({ usage: [] }) + expect(admission.admitKnowledgeDocuments).not.toHaveBeenCalled() + }) + + it('binds each occurrence to the stored tag value it shows', async () => { + await readDetailedKnowledgeTagUsage.execute({ + principal: chat, + input: { knowledgeBaseId: 'knowledge-b' }, + }) + const [, documents] = admission.admitKnowledgeDocuments.mock.calls[1] + expect(documents).toEqual([ + expect.objectContaining({ + id: 'doc-shared', + snapshot: { filename: 'shared.csv', tag2: 'ops' }, + }), + ]) + }) + + it('decides each (definition, document) occurrence on its own', async () => { + const result = await readDetailedKnowledgeTagUsage.execute({ + principal: chat, + input: { knowledgeBaseId: 'knowledge-b' }, + }) + + expect(result.usage[0]).toMatchObject({ + documentCount: 1, + documents: [{ id: 'doc-other' }], + }) + expect(result.usage[1]).toBe(detailed[1]) + }) + + it('leaves a person’s counts as stored, without reading documents', async () => { + const stats = [ + { + id: 'def-1', + tagSlot: 'tag1', + displayName: 'Region', + fieldType: 'text', + documentCount: 2, + chunkCount: 9, + }, + ] + mocks.readUsage.mockResolvedValue(stats) + + const result = await readKnowledgeTagUsage.execute({ + principal: sessionPrincipal, + input: { knowledgeBaseId: 'knowledge-b' }, + }) + + expect(result.usage).toBe(stats) + expect(mocks.readDetailedUsage).not.toHaveBeenCalled() + }) + }) }) diff --git a/apps/sim/lib/knowledge/application/tags.ts b/apps/sim/lib/knowledge/application/tags.ts index 9bf78e6f0e9..6371a8ced93 100644 --- a/apps/sim/lib/knowledge/application/tags.ts +++ b/apps/sim/lib/knowledge/application/tags.ts @@ -1,8 +1,10 @@ import { AuditAction, AuditResourceType } from '@sim/audit' import type { Principal } from '@sim/auth/principal' import { getPostgresConstraintName, getPostgresErrorCode } from '@sim/utils/errors' +import { admitContent } from '@/lib/core/application' import { OrchestrationError } from '@/lib/core/orchestration/types' import { generateRequestId } from '@/lib/core/utils/request' +import { admitKnowledgeDocuments } from '@/lib/knowledge/application/admit-knowledge-content' import { defineAuthorizedKnowledgeUseCase } from '@/lib/knowledge/application/authorized-knowledge-use-case' import { resolveActiveKnowledgeResourceContext, @@ -349,6 +351,37 @@ export const deleteKnowledgeTag = defineAuthorizedKnowledgeUseCase({ }), }) +/** + * The documents each tag definition lists, admitted per (definition, document) occurrence: a + * document shows a different tag value under each definition, so one occurrence's verdict says + * nothing about another's. + */ +async function admitTagUsage( + principal: Principal, + usage: Awaited> +): Promise>> { + /** A base with no tag definitions returns nothing, which is still an accounting. */ + if (usage.length === 0) await admitContent(principal, []) + const admitted = [] + for (const definition of usage) { + const ids = await admitKnowledgeDocuments( + principal, + definition.documents.map((doc) => ({ + id: doc.id, + value: doc, + snapshot: { filename: doc.name, [definition.tagSlot]: doc.tagValue }, + })) + ) + const documents = definition.documents.filter((doc) => ids.has(doc.id)) + admitted.push( + documents.length === definition.documents.length + ? definition + : { ...definition, documents, documentCount: documents.length } + ) + } + return admitted +} + export const readKnowledgeTagUsage = defineAuthorizedKnowledgeUseCase({ operation: knowledgeOperations.readTagUsage, resolveContext: ({ principal, input }: { principal: Principal; input: ListKnowledgeTagsInput }) => @@ -364,10 +397,9 @@ export const readDetailedKnowledgeTagUsage = defineAuthorizedKnowledgeUseCase({ operation: knowledgeOperations.readDetailedTagUsage, resolveContext: ({ principal, input }: { principal: Principal; input: ListKnowledgeTagsInput }) => resolveActiveKnowledgeResourceContext(input, principal), - async execute({ context }) { - return { - usage: await getTagUsage(context.knowledgeBaseId, generateRequestId(), context.access), - } + async execute({ principal, context }) { + const usage = await getTagUsage(context.knowledgeBaseId, generateRequestId(), context.access) + return { usage: await admitTagUsage(principal, usage) } }, }) diff --git a/apps/sim/lib/knowledge/documents/document-processor.ts b/apps/sim/lib/knowledge/documents/document-processor.ts index eacda690ff2..94546645e74 100644 --- a/apps/sim/lib/knowledge/documents/document-processor.ts +++ b/apps/sim/lib/knowledge/documents/document-processor.ts @@ -79,6 +79,7 @@ import type { VisionExtractionUsage } from '@/lib/knowledge/documents/vision-che import { assertKnowledgeOpaqueModelInputSafe, getKnowledgeOpaqueModelInputRegistry, + projectKnowledgeModelInput, } from '@/lib/knowledge/model-input-provenance' import { getFileExtension, isInternalFileUrl } from '@/lib/uploads/utils/file-utils' import { @@ -305,7 +306,7 @@ export async function processDocument( ? { content: image.renderedText, processingMethod: 'vision-extract' } : await parseDocument(fileUrl, filename, mimeType, access, workspaceId) access.signal?.throwIfAborted() - const { content, processingMethod, cloudUrl } = parseResult + const { content: parsedContent, processingMethod, cloudUrl } = parseResult if (parseResult.metadata?.detectedType || parseResult.metadata?.warning) { logger.info('Parser reported a warning for the document', { filename, @@ -320,13 +321,15 @@ export async function processDocument( * document that reports success while holding nothing. Failing here keeps it * visible with a reason instead. */ - if (parseResult.metadata?.degraded || !content.trim()) { + if (parseResult.metadata?.degraded || !parsedContent.trim()) { throw new PermanentDocumentProcessingError( 'no_extractable_text', unreadableDocumentMessage(filename) ) } + // Chunkers transform text; source and parser budgets already bound this materialized input. + const content = projectKnowledgeModelInput(parsedContent, Number.POSITIVE_INFINITY) let chunks: Chunk[] const metadata: FileParseMetadata = parseResult.metadata ?? {} @@ -398,7 +401,7 @@ export async function processDocument( } access.signal?.throwIfAborted() - const characterCount = content.length + const characterCount = parsedContent.length const tokenCount = chunks.reduce((sum, chunk) => sum + chunk.tokenCount, 0) logger.info(`Document processed: ${chunks.length} chunks, ${tokenCount} tokens`) diff --git a/apps/sim/lib/knowledge/documents/ocr-checkpoints.test.ts b/apps/sim/lib/knowledge/documents/ocr-checkpoints.test.ts index 9eddddb08ab..43498f3d722 100644 --- a/apps/sim/lib/knowledge/documents/ocr-checkpoints.test.ts +++ b/apps/sim/lib/knowledge/documents/ocr-checkpoints.test.ts @@ -12,7 +12,8 @@ const hoisted = vi.hoisted(() => ({ executeOcr: vi.fn(), })) vi.mock('@/lib/uploads/core/storage-service', () => storageServiceMock) -vi.mock('@/lib/knowledge/model-input-provenance', () => ({ +vi.mock(import('@/lib/knowledge/model-input-provenance'), async (importOriginal) => ({ + ...(await importOriginal()), getKnowledgeOpaqueModelInputRegistry: hoisted.provenance, assertKnowledgeOpaqueModelInputSafe: hoisted.provenance, })) diff --git a/apps/sim/lib/knowledge/model-input-provenance.ts b/apps/sim/lib/knowledge/model-input-provenance.ts index bf15f898be2..fa0a169dd16 100644 --- a/apps/sim/lib/knowledge/model-input-provenance.ts +++ b/apps/sim/lib/knowledge/model-input-provenance.ts @@ -116,12 +116,12 @@ export function getKnowledgeOpaqueModelInputRegistry(): ResolvedSecretTraceRegis return context.registry ?? new ResolvedSecretTraceRegistry() } -/** Projects one string immediately before it enters an embedding or reranking request. */ -export function projectKnowledgeModelInput(value: string): string { +/** Projects known literals in text before its transformation or delivery to a model. */ +export function projectKnowledgeModelInput(value: string, maxBytes?: number): string { const registry = knowledgeModelInputContext.getStore()?.registry if (!registry) return value - const projection = projectResolvedSecretModelContent(value, registry) + const projection = projectResolvedSecretModelContent(value, registry, maxBytes) if (!projection.safe || typeof projection.value !== 'string') { refuseResolvedSecretProjection({ site: 'knowledge.modelInput', diff --git a/apps/sim/lib/knowledge/secret-provenance.test.ts b/apps/sim/lib/knowledge/secret-provenance.test.ts index b16e7edecd2..0c8c3a37b62 100644 --- a/apps/sim/lib/knowledge/secret-provenance.test.ts +++ b/apps/sim/lib/knowledge/secret-provenance.test.ts @@ -5,9 +5,11 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' import type { DbTransaction } from '@/lib/db/types' import type { DurableSecretProvenance } from '@/lib/execution/durable-secret-provenance' import { + classifyKnowledgeDocumentMetadata, createKnowledgeDocumentSourceValue, importKnowledgePersistedResponseSecretProvenance, importKnowledgeSearchResultSecretProvenance, + knowledgeDocumentSnapshot, loadKnowledgeDocumentSecretRegistry, replaceKnowledgeDocumentSecretProvenanceInTx, } from '@/lib/knowledge/secret-provenance' @@ -167,3 +169,50 @@ describe('knowledge durable provenance enforcement', () => { expect(registry.isPermanentlyIncomplete()).toBe(true) }) }) + +describe('document metadata classification bound to the shown snapshot', () => { + const NUMBERED_ROW = { ...DOCUMENT_ROW, tag1: 'emea', number1: 42 } + + beforeEach(() => { + resetDbChainMock() + }) + + it('keeps the stored classification while the shown metadata is what is stored', async () => { + queueTableRows(document, [NUMBERED_ROW]) + const result = await classifyKnowledgeDocumentMetadata( + [NUMBERED_ROW.id], + new Map([[NUMBERED_ROW.id, [{ filename: 'source.pdf', tag1: 'emea', number1: '42' }]]]) + ) + expect(result.get(NUMBERED_ROW.id)?.status).toBe('exact') + }) + + it.each([ + ['a renamed document', { filename: 'previous-name.pdf' }], + ['an edited tag', { tag1: 'previous value' }], + ['a tag cleared since', { tag2: 'previous value' }], + ['an edited number', { number1: '7' }], + ])( + 'reads %s as unknown, so stale values are not paired with new provenance', + async (_, shown) => { + queueTableRows(document, [NUMBERED_ROW]) + const result = await classifyKnowledgeDocumentMetadata( + [NUMBERED_ROW.id], + new Map([[NUMBERED_ROW.id, [shown]]]) + ) + expect(result.get(NUMBERED_ROW.id)).toEqual({ status: 'unknown' }) + } + ) + + it('picks only the editable metadata a returned value shows', () => { + expect( + knowledgeDocumentSnapshot({ + id: 'd', + filename: 'f.pdf', + tag3: 'x', + fileUrl: '/u', + date1: 'd', + }) + ).toEqual({ filename: 'f.pdf', tag3: 'x' }) + expect(knowledgeDocumentSnapshot({ id: 'd', fileUrl: '/u' })).toBeUndefined() + }) +}) diff --git a/apps/sim/lib/knowledge/secret-provenance.ts b/apps/sim/lib/knowledge/secret-provenance.ts index 3f73183cb46..60cf942f339 100644 --- a/apps/sim/lib/knowledge/secret-provenance.ts +++ b/apps/sim/lib/knowledge/secret-provenance.ts @@ -6,7 +6,7 @@ import { embeddingSecretProvenance, } from '@sim/db/schema' import { sha256Hex } from '@sim/security/hash' -import { and, eq, inArray, isNull, type SQL } from 'drizzle-orm' +import { and, eq, inArray, isNull, type SQL, sql } from 'drizzle-orm' import type { DbTransaction } from '@/lib/db/types' import { bindDurableSecretProvenanceToValue, @@ -184,6 +184,8 @@ interface KnowledgeEmbeddingSecretProvenanceRow { const KNOWLEDGE_DOCUMENT_PROVENANCE_SELECTION = { id: document.id, + knowledgeBaseId: document.knowledgeBaseId, + storageKey: document.storageKey, secretProvenanceVersion: document.secretProvenanceVersion, filename: document.filename, fileUrl: document.fileUrl, @@ -222,9 +224,18 @@ const KNOWLEDGE_EMBEDDING_PROVENANCE_SELECTION = { entries: embeddingSecretProvenance.entries, } -function selectKnowledgeDocumentProvenanceRows(where: SQL | undefined) { +function selectKnowledgeDocumentProvenanceRows( + where: SQL | undefined, + deferInlineContent = false +) { return db - .select(KNOWLEDGE_DOCUMENT_PROVENANCE_SELECTION) + .select({ + ...KNOWLEDGE_DOCUMENT_PROVENANCE_SELECTION, + fileUrl: deferInlineContent + ? sql`CASE WHEN ${document.fileUrl} LIKE 'data:%' THEN '' ELSE ${document.fileUrl} END` + : document.fileUrl, + hasInlineFile: sql`${document.fileUrl} LIKE 'data:%'`, + }) .from(document) .leftJoin(documentSecretProvenance, eq(documentSecretProvenance.documentId, document.id)) .where(where) @@ -439,8 +450,180 @@ export async function loadKnowledgeDocumentSecretRegistry( return { registry, provenance, tracked: true } } +/** + * Metadata fields a person can edit in place that can hold text, which a returned snapshot is + * checked against. Date and boolean slots are typed columns no secret text can occupy. + */ +const EDITABLE_DOCUMENT_FIELDS = [ + 'filename', + 'sourceUrl', + 'tag1', + 'tag2', + 'tag3', + 'tag4', + 'tag5', + 'tag6', + 'tag7', + 'number1', + 'number2', + 'number3', + 'number4', + 'number5', +] as const satisfies readonly (keyof KnowledgeDocumentSourceValue)[] + +/** What a result shows of a document's metadata by stored field name, to bind its class to. */ +export type KnowledgeDocumentSnapshot = Readonly> + +const isEditableField = (field: string): boolean => + (EDITABLE_DOCUMENT_FIELDS as readonly string[]).includes(field) + +function comparableField(value: unknown): string | null { + if (value === null || value === undefined || value === '') return null + return String(value) +} + +/** A number slot is compared by value, since its text form can differ from its stored form. */ +function sameField(stored: unknown, shown: unknown): boolean { + const left = comparableField(stored) + const right = comparableField(shown) + if (left === right) return true + if (left === null || right === null) return false + return typeof stored === 'number' && Number(right) === stored +} + +function matchesDocumentSnapshot( + source: KnowledgeDocumentSourceValue, + snapshot: KnowledgeDocumentSnapshot +): boolean { + return Object.entries(snapshot).every( + ([field, shown]) => !isEditableField(field) || sameField(Reflect.get(source, field), shown) + ) +} + +/** Picks the editable metadata a returned value shows, when it shows any. */ +export function knowledgeDocumentSnapshot(value: unknown): KnowledgeDocumentSnapshot | undefined { + if (!value || typeof value !== 'object' || Array.isArray(value)) return undefined + const snapshot: Record = {} + let shown = false + for (const field of EDITABLE_DOCUMENT_FIELDS) { + if (Object.hasOwn(value, field)) { + snapshot[field] = Reflect.get(value, field) + shown = true + } + } + return shown ? snapshot : undefined +} + +/** + * Classifies the rendered metadata of each document. A document whose stored metadata no longer + * matches a snapshot the result shows reads as unknown: a concurrent edit must not pair the old + * values, which the result still carries, with the new row's classification. Inline originals + * needed for tracked bindings are loaded one at a time, never materialized as a metadata page. + */ +export async function classifyKnowledgeDocumentMetadata( + documentIds: readonly string[], + snapshots?: ReadonlyMap +): Promise> { + const ids = [...new Set(documentIds)] + const rows = ids.length + ? await selectKnowledgeDocumentProvenanceRows(inArray(document.id, ids), true) + : [] + const result = new Map() + for (const metadata of rows) { + const row = + metadata.hasInlineFile && metadata.secretProvenanceVersion !== null + ? (await selectKnowledgeDocumentProvenanceRows(eq(document.id, metadata.id)).limit(1))[0] + : metadata + if (!row) { + result.set(metadata.id, { status: 'unknown' }) + continue + } + const source = createKnowledgeDocumentSourceValue(row) + const stale = (snapshots?.get(row.id) ?? []).some( + (snapshot) => !matchesDocumentSnapshot(source, snapshot) + ) + result.set( + row.id, + stale + ? { status: 'unknown' } + : filterKnowledgeDocumentMetadataSecretProvenance( + readBoundKnowledgeDocumentSecretProvenance({ ...row, source }), + source + ) + ) + } + for (const id of ids) if (!result.has(id)) result.set(id, { status: 'unknown' }) + return result +} + +interface KnowledgeChunkContentSnapshot { + id: string + documentId: string + content: string +} + +/** Classifies chunk bytes against their current stored identity and content binding. */ +export async function classifyKnowledgeChunkContent( + chunks: readonly KnowledgeChunkContentSnapshot[] +): Promise> { + const ids = [...new Set(chunks.map((chunk) => chunk.id))] + const rows = ids.length + ? await selectKnowledgeEmbeddingProvenanceRows(inArray(embedding.id, ids)) + : [] + const rowById = new Map(rows.map((row) => [row.id, row])) + return new Map( + chunks.map((chunk): [string, DurableSecretProvenance] => { + const row = rowById.get(chunk.id) + return [ + chunk.id, + !row || row.documentId !== chunk.documentId || row.content !== chunk.content + ? { status: 'unknown' } + : readBoundKnowledgeEmbeddingSecretProvenance(row), + ] + }) + ) +} + +/** + * Classifies a result's chunk bytes and the document metadata it displays. Both are bound to + * the returned snapshots; tracked inline originals are loaded individually for metadata bindings. + */ +export async function classifyKnowledgeChunks( + chunks: readonly (KnowledgeChunkContentSnapshot & { snapshot?: KnowledgeDocumentSnapshot })[] +): Promise> { + const snapshots = new Map() + for (const chunk of chunks) { + if (chunk.snapshot) { + const current = snapshots.get(chunk.documentId) + if (current) current.push(chunk.snapshot) + else snapshots.set(chunk.documentId, [chunk.snapshot]) + } + } + const [contents, documents] = await Promise.all([ + classifyKnowledgeChunkContent(chunks), + classifyKnowledgeDocumentMetadata( + chunks.map((chunk) => chunk.documentId), + snapshots + ), + ]) + return new Map( + chunks.map((chunk): [string, DurableSecretProvenance] => [ + chunk.id, + mergeDurableSecretProvenance( + contents.get(chunk.id) ?? { status: 'unknown' }, + documents.get(chunk.documentId) ?? { status: 'unknown' } + ), + ]) + ) +} + /** Loads the bound document classification for copy/re-entry callers that do not need a registry. */ -export async function loadKnowledgeDocumentDurableSecretProvenance(documentId: string): Promise<{ +export async function loadKnowledgeDocumentDurableSecretProvenance( + documentId: string, + snapshot?: KnowledgeDocumentSnapshot +): Promise<{ + knowledgeBaseId: string + storageKey: string | null source: KnowledgeDocumentSourceValue provenance: DurableSecretProvenance tracked: boolean @@ -449,8 +632,13 @@ export async function loadKnowledgeDocumentDurableSecretProvenance(documentId: s if (!row) throw new Error('Document not found') const source = createKnowledgeDocumentSourceValue(row) return { + knowledgeBaseId: row.knowledgeBaseId, + storageKey: row.storageKey, source, - provenance: readBoundKnowledgeDocumentSecretProvenance({ ...row, source }), + provenance: + snapshot && !matchesDocumentSnapshot(source, snapshot) + ? { status: 'unknown' } + : readBoundKnowledgeDocumentSecretProvenance({ ...row, source }), tracked: row.secretProvenanceVersion === 1, } } diff --git a/apps/sim/lib/knowledge/transfer/export-archive.test.ts b/apps/sim/lib/knowledge/transfer/export-archive.test.ts index d087ec59bea..7bee0ddafdc 100644 --- a/apps/sim/lib/knowledge/transfer/export-archive.test.ts +++ b/apps/sim/lib/knowledge/transfer/export-archive.test.ts @@ -23,6 +23,7 @@ import { MAX_DOCUMENT_CHUNKS } from '@/lib/knowledge/documents/document-processi import { buildKnowledgeBundleArchive, knowledgeBundleFileName, + openKnowledgeExportFile, } from '@/lib/knowledge/transfer/export-archive' import type { ExportableChunk, ExportableDocument } from '@/lib/knowledge/transfer/export-source' @@ -82,6 +83,10 @@ function bundle(overrides: Partial = {}): KnowledgeBa }), exportableDocument({ id: TEXT_ONLY_ID, filename: 'wiki page', file: null }), ], + file: (document) => { + if (!document.file) throw new Error('Fixture has no original') + return openKnowledgeExportFile(document.file) + }, chunks: (documentId) => documentId === STORED_ID ? chunksOf(chunk(0, [0.25, 0.5]), chunk(1, [1, 2])) diff --git a/apps/sim/lib/knowledge/transfer/export-archive.ts b/apps/sim/lib/knowledge/transfer/export-archive.ts index 1185abb6d55..1a8a2f97f34 100644 --- a/apps/sim/lib/knowledge/transfer/export-archive.ts +++ b/apps/sim/lib/knowledge/transfer/export-archive.ts @@ -37,12 +37,16 @@ export function knowledgeBundleFileName(knowledgeBaseName: string): string { return `${safeBundleLeafName(knowledgeBaseName)}.simkb.zip` } -async function openFileSource(source: ExportableFileSource): Promise { +/** Opens one captured original; inline bytes can be supplied from their classified row snapshot. */ +export async function openKnowledgeExportFile( + source: ExportableFileSource, + inlineFileUrl?: string +): Promise { if (source.kind === 'storage') { return downloadFileStream({ key: source.key, context: 'knowledge-base' }) } return decodeDataUriWithinLimit( - await readInlineFileUrl(source.knowledgeBaseId, source.documentId), + inlineFileUrl ?? (await readInlineFileUrl(source.knowledgeBaseId, source.documentId)), MAX_KNOWLEDGE_DOCUMENT_FILE_SIZE ).buffer } @@ -149,7 +153,7 @@ async function appendBundleEntries( for (const document of bundle.documents) { const entries = bundleEntryPaths(document) if (document.file && entries.file) { - await appendEntry(archive, await openFileSource(document.file), entries.file, closed) + await appendEntry(archive, await bundle.file(document), entries.file, closed) } const chunkCount = entries.chunks ? await appendChunkEntry( diff --git a/apps/sim/lib/knowledge/transfer/export-source.ts b/apps/sim/lib/knowledge/transfer/export-source.ts index c2e5ca0844a..def2ee496a8 100644 --- a/apps/sim/lib/knowledge/transfer/export-source.ts +++ b/apps/sim/lib/knowledge/transfer/export-source.ts @@ -185,13 +185,15 @@ export async function listExportableDocuments( export async function* iterateDocumentChunks( knowledgeBaseId: string, documentId: string, - dimensions: KbEmbeddingDimensions | null + dimensions: KbEmbeddingDimensions | null, + admitPage?: (chunks: readonly (ExportableChunk & { id: string })[]) => Promise ): AsyncGenerator { const pageSize = dimensions ? CHUNK_PAGE_SIZE.vectors : CHUNK_PAGE_SIZE.text let after: number | null = null for (;;) { const page = await db .select({ + id: embedding.id, index: embedding.chunkIndex, content: embedding.content, tokenCount: embedding.tokenCount, @@ -211,7 +213,8 @@ export async function* iterateDocumentChunks( .orderBy(asc(embedding.chunkIndex)) .limit(pageSize) - yield* page + await admitPage?.(page) + for (const { id: _id, ...chunk } of page) yield chunk if (page.length < pageSize) break after = page[page.length - 1].index } diff --git a/apps/sim/lib/logs/application/operations.ts b/apps/sim/lib/logs/application/operations.ts index 1f83b5f956f..1e67b4c8f06 100644 --- a/apps/sim/lib/logs/application/operations.ts +++ b/apps/sim/lib/logs/application/operations.ts @@ -24,6 +24,7 @@ export const logOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', ...LOG_READER_PRINCIPAL_POLICY, }), // permission-group-exempt: aggregate run counts carry no execution payload, so there is nothing here for a group to withhold @@ -33,6 +34,7 @@ export const logOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', principalKinds: [...PUBLIC_API_PRINCIPAL_KINDS, 'delegated'], delegatedServices: ['copilot'], }), @@ -43,6 +45,7 @@ export const logOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'none', + contentDelivery: 'none', ...LOG_READER_PRINCIPAL_POLICY, }), // permission-group-exempt: the executor reading its own run's snapshot mid-flight; refusing it would fail runs the group permits diff --git a/apps/sim/lib/mcp/application/operations.ts b/apps/sim/lib/mcp/application/operations.ts index 1f38e05b576..1253fb734de 100644 --- a/apps/sim/lib/mcp/application/operations.ts +++ b/apps/sim/lib/mcp/application/operations.ts @@ -44,6 +44,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), listManagedConnections: defineWorkspaceOperation({ @@ -59,6 +60,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'mcp_tools.use', + contentDelivery: 'none', ...DISCOVERY_PRINCIPAL_POLICY, }), executeTool: defineWorkspaceOperation({ @@ -66,6 +68,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'mcp_tools.use', + contentDelivery: 'none', ...EXECUTION_PRINCIPAL_POLICY, }), /** @@ -90,6 +93,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), /** @@ -106,6 +110,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), listWorkflowDeploymentTools: defineWorkspaceOperation({ @@ -114,6 +119,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), createWorkflowDeploymentServer: defineWorkspaceOperation({ @@ -122,6 +128,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), /** @@ -143,6 +150,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), deleteWorkflowDeploymentServer: defineWorkspaceOperation({ @@ -151,6 +159,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), deployWorkflowTool: defineWorkspaceOperation({ @@ -159,6 +168,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), undeployWorkflowTool: defineWorkspaceOperation({ @@ -167,6 +177,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'deploy.mcp', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), read: defineWorkspaceOperation({ @@ -175,6 +186,7 @@ export const mcpServerOperations = { minimumRole: 'read', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), create: defineWorkspaceOperation({ @@ -183,6 +195,7 @@ export const mcpServerOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), register: defineWorkspaceOperation({ @@ -191,6 +204,7 @@ export const mcpServerOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), update: defineWorkspaceOperation({ @@ -199,6 +213,7 @@ export const mcpServerOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), reconfigure: defineWorkspaceOperation({ @@ -207,6 +222,7 @@ export const mcpServerOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), /** Pointing a server at another host or path; deployed workflows pin servers by id. */ @@ -216,6 +232,7 @@ export const mcpServerOperations = { minimumRole: 'admin', workspaceApiKey: 'deny', capability: 'mcp_tools.use', + contentDelivery: 'none', ...HUMAN_PRINCIPAL_POLICY, }), delete: defineWorkspaceOperation({ @@ -224,6 +241,7 @@ export const mcpServerOperations = { minimumRole: 'write', workspaceApiKey: 'allow', capability: 'mcp_tools.use', + contentDelivery: 'none', ...ALL_PRINCIPAL_POLICY, }), } as const diff --git a/apps/sim/lib/media/ffmpeg-content.integration.ts b/apps/sim/lib/media/ffmpeg-content.integration.ts new file mode 100644 index 00000000000..9c4557e634d --- /dev/null +++ b/apps/sim/lib/media/ffmpeg-content.integration.ts @@ -0,0 +1,308 @@ +/** Real file versions, storage, and output lineage around the media-processing boundary. */ +import { mkdtempSync } from 'node:fs' +import { rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import path from 'node:path' +import * as audit from '@sim/audit' +import { db } from '@sim/db' +import { auditLog, organization, user, workspace } from '@sim/db/schema' +import { deleteWorkspaceFixture } from '@sim/db/testing/workspace-fixtures' +import { generateId } from '@sim/utils/id' +import { eq, inArray } from 'drizzle-orm' +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' +import { observeContentDelivery, observeContentOutsideModel } from '@/lib/core/application' +import { encryptSecret } from '@/lib/core/security/encryption' +import { + createKnowledgeAclFixtureIds, + seedKnowledgeAclFixture, +} from '@/lib/knowledge/__integration__/seed-source-access-fixture' +import type { FfmpegOperation, MediaFile } from '@/lib/media/ffmpeg' +import { createCopilotChatPrincipal } from '@/lib/mothership/auth/application-delegation' +import { createRegistryContentGate } from '@/lib/mothership/content-delivery/registry-gate' +import { ffmpegServerTool } from '@/lib/mothership/tools/server/media/ffmpeg' +import * as workspaceReader from '@/lib/uploads/contexts/workspace' +import { + fetchWorkspaceFileBuffer, + getWorkspaceFile, + updateWorkspaceFileContent, + uploadWorkspaceFile, +} from '@/lib/uploads/contexts/workspace/workspace-file-manager' +import { + getBoundWorkspaceFileSecretProvenance, + type WorkspaceFileSecretProvenance, +} from '@/lib/uploads/contexts/workspace/workspace-file-secret-provenance' +import { WORKSPACE_FILES_DELEGATION_AUDIENCE } from '@/lib/workspace-files/application/authorization' +import { readWorkspaceFileContent } from '@/lib/workspace-files/application/read-workspace-file-content' +import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry' + +const storage = vi.hoisted(() => ({ root: '' })) +vi.mock('@/lib/uploads/core/setup.server', () => ({ + get UPLOAD_DIR_SERVER() { + return storage.root + }, +})) +vi.mock('@/lib/media/ffmpeg', () => ({ + runFfmpegOperation: async (_operation: FfmpegOperation, inputs: MediaFile[]) => ({ + buffer: inputs[0].buffer, + ext: 'bin', + contentType: 'application/octet-stream', + }), +})) + +const ids = createKnowledgeAclFixtureIds() +const SECRET = 'fixture-media-token-847159' +const CLEAN = { status: 'exact', entries: [] } as const +let known: WorkspaceFileSecretProvenance +let submittedAudits = 0 + +async function drainAudits() { + await vi.waitFor(async () => + expect( + await db + .select({ id: auditLog.id }) + .from(auditLog) + .where(eq(auditLog.workspaceId, ids.workspaceId)) + ).toHaveLength(submittedAudits) + ) +} + +async function replace(fileId: string, content: string, provenance: WorkspaceFileSecretProvenance) { + return updateWorkspaceFileContent( + ids.workspaceId, + fileId, + ids.aliceId, + Buffer.from(content), + undefined, + { + version: { source: 'api', authorUserId: ids.aliceId }, + secretProvenancePolicy: { mode: 'replace', provenance }, + } + ) +} + +beforeAll(async () => { + storage.root = mkdtempSync(path.join(tmpdir(), 'sim-media-lineage-')) + await seedKnowledgeAclFixture(ids) + known = { + status: 'exact', + entries: [ + { + name: 'MEDIA_TOKEN', + encryptedValue: (await encryptSecret(SECRET)).encrypted, + sourceUserId: ids.aliceId, + sourceWorkspaceId: ids.workspaceId, + }, + ], + } + const record = audit.recordAudit + vi.spyOn(audit, 'recordAudit').mockImplementation((entry) => { + if (entry.workspaceId === ids.workspaceId) submittedAudits += 1 + record(entry) + }) + const batch = audit.recordAuditBatch + vi.spyOn(audit, 'recordAuditBatch').mockImplementation((entries) => { + submittedAudits += entries.filter((entry) => entry.workspaceId === ids.workspaceId).length + batch(entries) + }) +}) + +afterAll(async () => { + await drainAudits() + vi.restoreAllMocks() + await db.delete(auditLog).where(eq(auditLog.workspaceId, ids.workspaceId)) + await deleteWorkspaceFixture(db, eq(workspace.id, ids.workspaceId)) + await db.delete(organization).where(eq(organization.id, ids.organizationId)) + await db.delete(user).where(inArray(user.id, [ids.aliceId, ids.bobId])) + await rm(storage.root, { recursive: true, force: true }) +}) + +describe('media input version lineage', () => { + it.each(['known', 'unknown'] as const)( + 'preserves the storage size-limit error for an admitted %s media source', + async (classification) => { + const file = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from(SECRET), + `bounded-${generateId()}.bin`, + 'application/octet-stream', + { + secretProvenance: classification === 'known' ? known : { status: 'unknown' }, + notifyWorkspaceChange: false, + } + ) + await expect( + observeContentOutsideModel(() => + readWorkspaceFileContent.execute({ + principal: createCopilotChatPrincipal( + { userId: ids.aliceId, workspaceId: ids.workspaceId }, + WORKSPACE_FILES_DELEGATION_AUDIENCE + ), + input: { fileId: file.id, maxBytes: 8, includeSecretProvenance: true }, + }) + ) + ).rejects.toMatchObject({ + name: 'PayloadSizeLimitError', + maxBytes: 8, + observedBytes: Buffer.byteLength(SECRET), + }) + } + ) + + it.each(['known', 'unknown'] as const)( + 'does not expose storage failures for %s input bytes before a read result exists', + async (classification) => { + const file = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from(SECRET), + `${SECRET}-${classification}.bin`, + 'application/octet-stream', + { + secretProvenance: classification === 'known' ? known : { status: 'unknown' }, + notifyWorkspaceChange: false, + } + ) + await rm(path.join(storage.root, file.key)) + await expect(fetchWorkspaceFileBuffer(file, { maxBytes: 1024 })).rejects.toThrow(SECRET) + const result = await ffmpegServerTool.execute( + { operation: 'convert', inputs: { files: [{ path: file.id }] } }, + { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + toolCallId: generateId(), + copilotToolExecution: true, + resolvedSecretTraceRegistry: new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + } + ) + expect(result.success).toBe(false) + expect(result.message).not.toContain(SECRET) + expect(result.message).not.toContain(storage.root) + await expect( + observeContentDelivery( + createRegistryContentGate({ + registry: new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + actorUserId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + () => + readWorkspaceFileContent.execute({ + principal: createCopilotChatPrincipal( + { userId: ids.aliceId, workspaceId: ids.workspaceId }, + WORKSPACE_FILES_DELEGATION_AUDIENCE + ), + input: { fileId: file.id }, + }) + ) + ).rejects.toThrow( + classification === 'known' + ? 'File content could not be read' + : 'File cannot be sent to a model because its secret provenance is unavailable' + ) + await expect( + readWorkspaceFileContent.execute({ + principal: { kind: 'session', userId: ids.aliceId, sessionId: generateId() }, + input: { fileId: file.id, includeSecretProvenance: true }, + }) + ).rejects.toThrow(SECRET) + } + ) + + it.each(['known', 'unknown'] as const)( + 'persists the %s evidence of the bytes read after reference resolution', + async (classification) => { + const file = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from('public version one'), + `input-${generateId()}.bin`, + 'application/octet-stream', + { secretProvenance: CLEAN, notifyWorkspaceChange: false } + ) + const provenance = classification === 'known' ? known : { status: 'unknown' as const } + const execute = readWorkspaceFileContent.execute + const pause = vi + .spyOn(readWorkspaceFileContent, 'execute') + .mockImplementationOnce(async (args) => { + await replace(file.id, SECRET, provenance) + return execute(args) + }) + try { + const result = await ffmpegServerTool.execute( + { + operation: 'convert', + inputs: { files: [{ path: `files/${file.name}` }] }, + outputs: { files: [{ path: `files/output-${generateId()}.bin` }] }, + }, + { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + toolCallId: generateId(), + copilotToolExecution: true, + resolvedSecretTraceRegistry: new ResolvedSecretTraceRegistry([], { + userId: ids.aliceId, + workspaceId: ids.workspaceId, + }), + } + ) + expect(result.success, result.message).toBe(true) + if (!result.fileId) throw new Error('No output file was persisted') + const output = await getWorkspaceFile(ids.workspaceId, result.fileId) + if (!output) throw new Error('Output file cannot be read back') + expect((await fetchWorkspaceFileBuffer(output, { maxBytes: output.size })).toString()).toBe( + SECRET + ) + expect( + await getBoundWorkspaceFileSecretProvenance(ids.workspaceId, { + fileId: output.id, + key: output.key, + context: output.storageContext ?? 'workspace', + contentUpdatedAt: output.contentUpdatedAt ?? undefined, + }) + ).toEqual(provenance) + } finally { + pause.mockRestore() + } + } + ) + + it('returns the captured version evidence when a replacement commits during its storage read', async () => { + const file = await uploadWorkspaceFile( + ids.workspaceId, + ids.aliceId, + Buffer.from(SECRET), + `input-${generateId()}.bin`, + 'application/octet-stream', + { secretProvenance: known, notifyWorkspaceChange: false } + ) + const fetchContent = workspaceReader.fetchWorkspaceFileBuffer + const pause = vi + .spyOn(workspaceReader, 'fetchWorkspaceFileBuffer') + .mockImplementationOnce(async (...args) => { + const bytes = await fetchContent(...args) + await replace(file.id, 'public replacement', CLEAN) + return bytes + }) + try { + const result = await readWorkspaceFileContent.execute({ + principal: { kind: 'session', userId: ids.aliceId, sessionId: generateId() }, + input: { + fileId: file.id, + assertedWorkspaceId: ids.workspaceId, + includeSecretProvenance: true, + }, + }) + expect(result.content.toString()).toBe(SECRET) + expect(result.file.key).toBe(file.key) + expect(result.secretProvenance).toEqual(known) + } finally { + pause.mockRestore() + } + }) +}) diff --git a/apps/sim/lib/mothership/__integration__/content-delivery-compat.baseline.json b/apps/sim/lib/mothership/__integration__/content-delivery-compat.baseline.json new file mode 100644 index 00000000000..2f85e7c1452 --- /dev/null +++ b/apps/sim/lib/mothership/__integration__/content-delivery-compat.baseline.json @@ -0,0 +1,875 @@ +{ + "GET /api/v2/workspaces": { + "status": 200, + "json": { + "data": [ + { + "color": "#33C482", + "createdAt": "