feat(data-retention): granular PII redaction stages (input + block outputs) by TheodoreSpeaks · Pull Request #5272 · simstudioai/sim

TheodoreSpeaks · 2026-06-29T22:20:09Z

Summary

Add two execution-altering PII redaction stages alongside the existing log redaction: redact the workflow input before execution, and mask every block output in-flight before the next block reads it
Per-stage policy (entity types + language) for each of Logs / Workflow input / Block outputs; resolved most-specific-wins per workspace, with full back-compat for existing logs-only rules
In-flight stages fail-fast (abort the run) on a Presidio error instead of scrubbing or leaking; the logs stage keeps scrub-to-marker
Reuse the shared HTTP → Presidio path; block-output redaction runs before payload compaction so offloaded large values are still masked
Settings UI: chip-tabs across the three stages, language-first picker with the entity grid filtered to that language's recognizers, and a confirmation before removing a workspace override

Type of Change

New feature

Testing

Tested manually. Unit tests for resolver back-compat, redactObjectStrings + failure modes, and the contract schema. bun run lint, check:api-validation:strict, and check:migrations origin/staging all pass.

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…tputs)

vercel · 2026-06-29T22:20:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated (UTC)
docs	Skipped		Jun 30, 2026 7:46pm

cursor · 2026-06-29T22:20:19Z

PR Summary

High Risk
Changes execution-time data (inputs, block outputs, streams, memory) and log persistence with fail-fast vs scrub semantics; misconfiguration or Presidio outages can abort runs or affect workflow correctness, not only observability.

Overview
Adds three independently configurable PII redaction stages (workflow input, block outputs, logs), each with its own entity types and language, while legacy flat rules still map to logs-only.

Runtime: Workflow input is masked before execution when the input stage is on. Block outputs are masked in-flight (before compaction and downstream blocks), including buffer-only streaming so raw chunks are not forwarded when that stage is enabled; policy propagates to child workflows and agent memory writes. Input/block stages abort the run on Presidio failure (onFailure: 'throw'); log persistence keeps scrub-to-marker behavior. Log redaction now uses only the logs stage, applies without the pii-redaction feature flag (stored rules are the source of truth), and hydrates large-value refs before masking so offloaded content gets the logs policy.

Presidio / batching: New /analyze_batch and /anonymize_batch endpoints; masking paths chunk by shared byte/count budgets and use batched analyze/anonymize instead of per-string concurrency.

Settings & contracts: Data retention UI uses stage tabs, language-filtered entity grids, and confirm-on-remove for overrides; API/schema accept stages with validation (enabled stages must pick at least one entity type).

^{Reviewed by Cursor Bugbot for commit f0c71cc. Bugbot is set up for automated code reviews on this repo. Configure here.}

greptile-apps · 2026-06-29T22:25:29Z

Greptile Summary

This PR adds granular PII redaction stages for workflow execution and logs. The main changes are:

Per-stage PII policies for workflow input, block outputs, and logs.
Input and block-output masking during workflow execution.
Batched Presidio masking paths for higher-volume redaction.
Settings UI updates for stage-specific entity and language selection.
Log redaction support for offloaded large values.

Confidence Score: 4/5

This is close, but the restored large-value path should be fixed before merging.

Normal execution now masks block outputs before compaction.
Log persistence now hydrates and masks offloaded large values.
Restored execution state can still carry refs to raw offloaded values into resumed blocks.

apps/sim/lib/workflows/executor/execution-core.ts

Security Review

Block-output redaction can still miss restored large-value refs from older paused or run-from-block snapshots, leaving raw PII reachable during resumed execution.

Important Files Changed

Filename	Overview
apps/sim/lib/workflows/executor/execution-core.ts	Adds execution-time policy resolution and restore masking, but restored large-value refs can still bypass block-output redaction.
apps/sim/executor/execution/block-executor.ts	Masks block outputs before compaction and prevents raw streaming chunks from being forwarded when block-output redaction is enabled.
apps/sim/lib/logs/execution/pii-large-values.ts	Adds hydrate, mask, and re-store handling for offloaded values in the log redaction path.
apps/sim/lib/billing/retention.ts	Resolves per-stage PII policy with stored rules as the execution-time source of truth.

_{Reviews (12): Last reviewed commit: "fix(data-retention): always apply logs p..." | Re-trigger Greptile}

…flows

TheodoreSpeaks · 2026-06-30T00:40:25Z

@greptile review

… resume

TheodoreSpeaks · 2026-06-30T00:46:12Z

@greptile review

…nconditionally (no fail-open)

TheodoreSpeaks · 2026-06-30T00:57:36Z

@greptile review

…edaction

TheodoreSpeaks · 2026-06-30T01:09:03Z

@greptile review

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

TheodoreSpeaks · 2026-06-30T17:14:06Z

@greptile review

TheodoreSpeaks · 2026-06-30T17:23:38Z

@greptile review

…no fail-open)

TheodoreSpeaks · 2026-06-30T17:34:59Z

@greptile review

… = off everywhere

…block-output redaction is off

TheodoreSpeaks · 2026-06-30T19:21:49Z

@greptile review

greptile-apps · 2026-06-30T19:24:03Z

+      // Limitation: this walks inline strings only — values offloaded to
+      // large-value storage are still refs here and are not re-masked. In the
+      // normal flow that is safe (a run with the stage on masks before offload);
+      // the gap is the narrow case of a run that offloaded a large value while
+      // the stage was OFF and is resumed after the stage is turned ON.


Large values bypass masking

When block-output redaction is enabled after a workflow already offloaded large block outputs, this restore path only masks inline strings in the snapshot. The offloaded payloads stay behind large-value refs. On resume or run-from-block, downstream blocks can still read the raw restored payload, and log persistence can skip the large-value scrub because block-output redaction is now enabled. This leaves raw PII reachable from prior block outputs after the stage is turned on.

cursor · 2026-06-30T19:25:51Z

          abortSignal: ctx.abortSignal,
+          // Propagate in-flight block-output redaction into child workflows so
+          // nested blocks mask outputs too (recurses: each child forwards it).
+          piiBlockOutputRedaction: ctx.piiBlockOutputRedaction,


Child workflows skip input redaction

Medium Severity

The new workflow-input PII stage runs only in executeWorkflowCore on top-level processedInput. Nested child runs are started with a direct Executor and pass childWorkflowInput unchanged. Only the block-output policy is forwarded on the context, so when the input stage is on and block outputs are off, mapped or explicit child input can execute and produce downstream state without in-flight input masking.

^{Reviewed by Cursor Bugbot for commit 8f86d77. Configure here.}

…logs (preserve redacted content)

TheodoreSpeaks · 2026-06-30T19:34:42Z

@greptile review

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 2 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 6e9587a. Configure here.}

greptile-apps · 2026-06-30T19:39:55Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay unmasked When a paused run or run-from-block snapshot contains a large-value ref that was created before block-output redaction was enabled, this call only masks inline strings. Large-value refs are treated as opaque by redactObjectStrings, so the ref still points at the original offloaded bytes. The later warm-up step can materialize that raw value for downstream blocks, letting them read or send unredacted PII even though the block-output stage is enabled.

…n logs stage is on

TheodoreSpeaks · 2026-06-30T19:46:37Z

@greptile review

greptile-apps · 2026-06-30T19:49:02Z

+        snapshot.state.blockStates = await redactObjectStrings(
+          snapshot.state.blockStates,
+          blockOutputOpts
+        )


Refs stay raw

This restore path still only masks inline strings. When a paused run or run-from-block snapshot contains a large-value ref created before block-output redaction was enabled, redactObjectStrings leaves the ref untouched. The later warm-up can materialize that original offloaded value for downstream blocks, so the resumed workflow can read raw PII even though block-output redaction is now enabled. This path needs to hydrate, mask, and re-store restored refs before downstream state can use them.

feat(data-retention): granular PII redaction stages (input + block ou…

f24d8cc

…tputs)

cursor Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

vercel Bot deployed to Preview June 29, 2026 22:24 View deployment

greptile-apps Bot reviewed Jun 29, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): propagate block-output redaction into child work…

36f2a3d

…flows

vercel Bot temporarily deployed to Preview June 30, 2026 00:37 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): close block-output redaction gaps on streaming +…

bb3a84b

… resume

vercel Bot temporarily deployed to Preview June 30, 2026 00:45 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/executor/execution/block-executor.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts Outdated

fix(data-retention): drain+mask streamed output, resolve PII policy u…

0b81fed

…nconditionally (no fail-open)

vercel Bot temporarily deployed to Preview June 30, 2026 00:56 Inactive

test(testing): support leftJoin().where().limit() in shared db mock

2d75987

vercel Bot temporarily deployed to Preview June 30, 2026 01:05 Inactive

fix(data-retention): mask agent/Pi memory writes under block-output r…

eb6b25a

…edaction

vercel Bot temporarily deployed to Preview June 30, 2026 01:08 Inactive

Merge remote-tracking branch 'origin/staging' into feat/pii-granular-…

324b04c

…redaction # Conflicts: # apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/app/api/organizations/[id]/data-retention/route.ts

vercel Bot deployed to Preview June 30, 2026 17:01 View deployment

fix(data-retention): guard partial PII stages in GET normalize

d55b557

vercel Bot temporarily deployed to Preview June 30, 2026 17:06 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/guardrails/validate_pii.ts

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/retention.ts

fix(guardrails): fail closed on misaligned Presidio batch responses

a911af8

vercel Bot temporarily deployed to Preview June 30, 2026 17:23 Inactive

fix(data-retention): enabled stage with no entity types redacts all (…

31f2e3f

…no fail-open)

vercel Bot temporarily deployed to Preview June 30, 2026 17:34 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/retention.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/ee/data-retention/components/data-retention-settings.tsx

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/workflows/executor/execution-core.ts

fix(data-retention): reject enabled stage with no entity types; empty…

437d2bb

… = off everywhere

vercel Bot temporarily deployed to Preview June 30, 2026 17:47 Inactive

docs(data-retention): note resume remask covers inline values only

78b2c56

vercel Bot temporarily deployed to Preview June 30, 2026 17:49 Inactive

fix(data-retention): scrub offloaded large-value refs from logs when …

8f86d77

…block-output redaction is off

vercel Bot temporarily deployed to Preview June 30, 2026 19:21 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

cursor Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): hydrate, mask, and re-store large-value refs in …

6e9587a

…logs (preserve redacted content)

vercel Bot temporarily deployed to Preview June 30, 2026 19:34 Inactive

cursor Bot reviewed Jun 30, 2026

View reviewed changes

Comment thread apps/sim/lib/logs/execution/logger.ts Outdated

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

fix(data-retention): always apply logs policy to large-value refs whe…

f0c71cc

…n logs stage is on

vercel Bot temporarily deployed to Preview June 30, 2026 19:46 Inactive

greptile-apps Bot reviewed Jun 30, 2026

View reviewed changes

Uh oh!

Conversation

TheodoreSpeaks commented Jun 29, 2026

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cursor Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Summary

Uh oh!

Uh oh!

greptile-apps Bot commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 4/5

Security Review

Important Files Changed

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Jun 30, 2026

Choose a reason for hiding this comment

Child workflows skip input redaction

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

TheodoreSpeaks commented Jun 30, 2026

Uh oh!

greptile-apps Bot Jun 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Jun 29, 2026 •

edited

Loading

cursor Bot commented Jun 29, 2026 •

edited

Loading

greptile-apps Bot commented Jun 29, 2026 •

edited

Loading