improvement(provenance): aggregate and attribute unrecorded durable reads - #7017
Conversation
…eads Fail-open on unrecorded durable provenance rests on one compensating control: the audit entry telling the people who own the secrets that a read proceeded unvouched. An audit of all four surfaces found the control incomplete in exactly the places this closes, and confirmed the policy itself sound — so nothing here changes what any read or write does, only what gets recorded about it. Knowledge was the one surface with no audit trail at all: the per-record import reports without a workspace, and the report skips the audit row when it cannot name one, so fail-open knowledge reads emitted one error log line per record and zero audit entries. Both importers now count unrecorded records while the surface is open and report once per read with the workspace, actor, and count — the shape memory and tables already use. The search read reports once across chunks and rendered metadata, and only when the registry did not latch, since a latched read never reaches a model. Fault returns stay silent; those reads fail closed. Memory had the one silent local degrade: a record whose canonical hash outgrows its bounds, or whose entries fail normalization, was stored unknown with nothing logged anywhere — the table writer logs its equivalent. The binding now logs the cause at error where it is decided. An incoming unknown stays silent; its producer already reported. The memory list contract gains the page ceiling every other list already has (max 1000, matching the table convention); no caller in the repo passes a limit at all, and the route is internal-auth only. Workspace-file audit rows now carry the acting user where the caller already holds one — copilot vfs, the agent and mothership handlers, and the provider attachment filter. Everywhere else, including principals with no user to name, the actor stays null, which the report type has always permitted. Two comments catch up with the code: the file sidecar stores three statuses since the absence/taint split, and the mounted-file scanner's scan-overflow-to-taint is deliberate where the registry scan over-approximates — that scan only narrows an already-sound candidate set, while this one decides whether egress redaction would suffice for bytes the same matcher just failed on.
|
The latest updates on your projects. Learn more about Vercel for GitHub. |
PR SummaryMedium Risk Overview Knowledge now counts unrecorded documents/chunks and emits one Workspace files pass the acting user into unrecorded-read reports from copilot VFS, agent/mothership handlers, and provider attachment filtering; unnamed principals stay Memory logs the local exact→unknown degrade (hash overflow or unnormalizable entries) at write time. Reviewed by Cursor Bugbot for commit f0b531d. Configure here. |
Greptile SummaryThis PR improves durable-secret provenance observability without changing the existing fail-open and fail-closed read decisions.
Confidence Score: 5/5The PR appears safe to merge, with the changed reporting and attribution paths preserving the existing provenance enforcement behavior. The aggregate reports are emitted on successful fail-open reads, while faulted or incomplete registries continue to prevent unsafe provenance-bearing content from reaching model input; no actionable regression remains.
|
| Filename | Overview |
|---|---|
| apps/sim/lib/knowledge/secret-provenance.ts | Suppresses per-record knowledge reports, returns aggregate counts for search, and emits one attributed persisted-read report after successful imports. |
| apps/sim/lib/knowledge/application/search.ts | Combines unknown chunk and rendered-document metadata counts into one report while preserving fail-closed behavior for incomplete registries. |
| apps/sim/lib/uploads/contexts/workspace/workspace-file-secret-provenance.ts | Threads optional acting-user identity through unrecorded workspace-file reporting without changing classification decisions. |
| apps/sim/lib/memory/secret-provenance.ts | Adds an error log when exact incoming memory provenance degrades locally during hash binding or entry normalization. |
| apps/sim/lib/api/contracts/memory.ts | Adds the documented maximum of 1,000 records to the memory-list query contract. |
| apps/sim/providers/index.ts | Supplies the provider request user as the audit actor when filtering model-facing workspace-file attachments. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart LR
A[Durable knowledge, memory, or file read] --> B{Provenance status}
B -->|Exact| C[Import provenance]
B -->|Unrecorded and unenforced| D[Proceed fail-open]
D --> E[Aggregate affected records]
E --> F[Report workspace and acting user]
B -->|Unknown fault or enforced| G[Latch registry or refuse]
G --> H[Prevent unsafe model egress]
Reviews (1): Last reviewed commit: "improvement(provenance): aggregate and a..." | Re-trigger Greptile
1 participant
Summary
workspaceId,actorUserId, andaffectedCount— the same shape memory and tables already ship. Search reports once across chunks + rendered metadata, and only when the registry didn't latch; fault returns (row-missing/content-mismatch) stay silent because those reads fail closed.unknown— now logs its cause at error where it's decided, matching the table writer. Incomingunknownstays silent (its producer already reported).GET /api/memorylimitgains.max(1000)per the tables convention. Internal-auth route; no caller in the repo passes a limit at all.Deliberately not touched (audit-confirmed accepted limitations): the fail-open default and enforcement env, the memory whole-array hash binding (follow-up with its own migration review), exact-empty certification of non-runtime writes, binary-output taint, the <8-char literal floor, and the envelope bounds. No schema changes, no migrations, no new incompleteness reasons or causes.
Type of Change
Testing
2,313 tests green across every affected suite (knowledge, memory, uploads/workspace, copilot handlers/request, providers, agent/mothership handlers, api/memory, api/knowledge). New tests pin: one aggregated knowledge report per read with workspace/actor/count and none on fault or enforced paths; the search importer returning its count instead of reporting; the memory degrade log firing only for local degradation; the limit ceiling; the actor flowing into (and defaulting to null on) file audit rows.
bun run type-checkclean,bun run lintclean, all 33 audits pass includingcheck:api-validation:strict.Checklist