Skip to content

Commit 9c3aab4

Browse files
authored
fix(pii): mask texts over spaCy's length limit and drop encoded files from logs (#9979)
* fix(pii): keep encoded files away from Presidio and split texts too long for one NER pass * fix(pii): mask long texts inside Presidio and limit encoded-file replacement to the log - Presidio runs pattern and custom recognizers over the whole text and spaCy NER in overlapping windows when a text exceeds spaCy's max_length, so no match can straddle a cut; the app no longer splits texts - Encoded file bodies are replaced only in the execution log (explicit replaceEncodedBinary option set by the log entry points); live data keeps every string on the masking path - Data URL headers are parsed through the first comma, and log-mode detection no longer requires mixed character classes - e2e: setup failures are recorded in the report * fix(pii): keep context scoring for thresholded long texts and match data: case-insensitively - Long-text NER windows also run the pattern entities when a score_threshold is set, so context-boosted matches clear it as they do on short texts - The data: URI scheme is matched case-insensitively - e2e asserts each name is masked as one <PERSON> span and is registered as test:pii-redaction:e2e * fix(pii): keep every NER window's spans instead of assuming a neighbour found them NER reads context, so a window can find a name the previous one missed or cut short; spans are now only deduplicated by exact position, and overlapping spans are left to the anonymizer to merge
1 parent 4180246 commit 9c3aab4

10 files changed

Lines changed: 666 additions & 60 deletions

File tree

‎apps/docs/content/docs/platform/enterprise/data-retention.mdx‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,8 @@ Redaction applies to text strings at the configured workflow stages. It does not
118118

119119
If log redaction fails, affected strings are replaced with `[REDACTION_FAILED]` rather than stored unmasked. Failure at Workflow input or Block outputs stops the run instead of passing unredacted values downstream.
120120

121+
Base64-encoded file content, such as a `data:` URL or a long base64 string, can't be checked for PII as text, because any PII is inside the encoded file. In stored logs it is replaced with `[BINARY_CONTENT_REDACTED]`. At Workflow input and Block outputs it is masked like any other string. **Run from block** and re-running with a previous run's input read that run's log, so they see the marker in place of that content, the same way they see masked PII.
122+
121123
## Self-hosted setup
122124

123125
### Enable cleanup and schedule it

‎apps/pii/server.py‎

Lines changed: 122 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -184,6 +184,7 @@ def build_analyzer() -> AnalyzerEngine:
184184
anonymizer: AnonymizerEngine
185185
NER_ENTITIES: frozenset
186186
_BLANK_ARTIFACTS: dict
187+
_NLP_MAX_LENGTH: dict
187188

188189
# Every entity the spaCy NER recognizers can actually produce. A request touching
189190
# any of these must run spaCy; a request naming only non-NER (regex/checksum)
@@ -227,6 +228,96 @@ def _regex_only(entities: list[str] | None, score_threshold: float | None) -> bo
227228
)
228229

229230

231+
# spaCy refuses (error E088) any text longer than its model's `max_length`, so a
232+
# longer text runs NER in windows of this many characters, following spaCy's
233+
# guidance to segment long documents (it sizes NER memory at roughly 1GB per
234+
# 100,000 characters of one text).
235+
NER_WINDOW_CHARS = 100_000
236+
# How much of the previous window each window repeats. Far longer than any entity
237+
# spaCy tags, so an entity cut by one window's edge lies whole inside the next.
238+
NER_WINDOW_OVERLAP = 1_000
239+
240+
241+
def _exceeds_nlp_length(text: str, language: str) -> bool:
242+
limit = _NLP_MAX_LENGTH.get(language)
243+
return limit is not None and len(text) > limit
244+
245+
246+
def _window_entities(
247+
entities: list[str] | None, language: str, score_threshold: float | None
248+
) -> list[str]:
249+
"""The requested built-in entities this language serves that windows must find,
250+
or all of them when `entities` is empty (detect-all). Without a score_threshold
251+
that is only what spaCy NER produces; with one it also includes the pattern
252+
entities, whose context-boosted score can decide whether a match clears the
253+
threshold and needs the real NLP artifacts only a window has."""
254+
served = set(analyzer.get_supported_entities(language))
255+
if score_threshold is None or score_threshold <= 0:
256+
served &= NER_ENTITIES
257+
return sorted(served if not entities else served.intersection(entities))
258+
259+
260+
def _analyze_long(
261+
text: str,
262+
language: str,
263+
entities: list[str] | None,
264+
score_threshold: float | None,
265+
return_decision_process: bool = False,
266+
ad_hoc_recognizers: list[PatternRecognizer] | None = None,
267+
) -> list[RecognizerResult]:
268+
"""Analyze a text longer than spaCy's `max_length`.
269+
270+
Pattern, checksum, and custom recognizers have no length limit, so they run
271+
once over the whole text (against blank NLP artifacts, as the regex-only path
272+
does) and no match can straddle a cut. Only NER runs in overlapping windows
273+
(see NER_WINDOW_CHARS), asked for just its own entities, plus the pattern
274+
entities when a score_threshold makes context scoring matter (see
275+
`_window_entities`). Every window's spans are kept, shifted to whole-text
276+
offsets: NER reads context, so a window can find a name its neighbour missed
277+
or cut short. A span found more than once keeps its highest score, and
278+
overlapping spans of one type are merged by the anonymizer."""
279+
entities = entities or None
280+
results = analyzer.analyze(
281+
text=text,
282+
language=language,
283+
entities=entities,
284+
score_threshold=score_threshold,
285+
return_decision_process=return_decision_process,
286+
nlp_artifacts=_BLANK_ARTIFACTS.get(language),
287+
ad_hoc_recognizers=ad_hoc_recognizers or None,
288+
)
289+
window_entities = _window_entities(entities, language, score_threshold)
290+
if not window_entities:
291+
return results
292+
step = NER_WINDOW_CHARS - NER_WINDOW_OVERLAP
293+
for start in range(0, len(text), step):
294+
for result in analyzer.analyze(
295+
text=text[start : start + NER_WINDOW_CHARS],
296+
language=language,
297+
entities=window_entities,
298+
score_threshold=score_threshold,
299+
return_decision_process=return_decision_process,
300+
):
301+
results.append(
302+
RecognizerResult(
303+
result.entity_type,
304+
result.start + start,
305+
result.end + start,
306+
result.score,
307+
result.analysis_explanation,
308+
result.recognition_metadata,
309+
)
310+
)
311+
if start + NER_WINDOW_CHARS >= len(text):
312+
break
313+
unique: dict[tuple[str, int, int], RecognizerResult] = {}
314+
for result in results:
315+
key = (result.entity_type, result.start, result.end)
316+
if key not in unique or result.score > unique[key].score:
317+
unique[key] = result
318+
return list(unique.values())
319+
320+
230321
def _analyze_one(
231322
text: str,
232323
language: str,
@@ -235,12 +326,15 @@ def _analyze_one(
235326
return_decision_process: bool = False,
236327
ad_hoc_recognizers: list[PatternRecognizer] | None = None,
237328
):
329+
regex_only = _regex_only(entities, score_threshold)
330+
if not regex_only and _exceeds_nlp_length(text, language):
331+
return _analyze_long(
332+
text, language, entities, score_threshold, return_decision_process, ad_hoc_recognizers
333+
)
238334
# Regex-only requests reuse a blank NlpArtifacts to skip the spaCy NLP pass;
239335
# otherwise analyze() computes artifacts (runs spaCy) as usual. Custom-pattern
240336
# recognizers are regex-based, so they run fine against the blank artifacts.
241-
nlp_artifacts = (
242-
_BLANK_ARTIFACTS.get(language) if _regex_only(entities, score_threshold) else None
243-
)
337+
nlp_artifacts = _BLANK_ARTIFACTS.get(language) if regex_only else None
244338
return analyzer.analyze(
245339
text=text,
246340
language=language,
@@ -264,7 +358,9 @@ def _analyze_many(
264358
score_threshold: float | None,
265359
ad_hoc_recognizers: list[PatternRecognizer] | None = None,
266360
):
267-
"""Analyze many texts, skipping the spaCy pass for regex-only requests."""
361+
"""Analyze many texts, skipping the spaCy pass for regex-only requests and
362+
analyzing any text longer than spaCy's `max_length` on its own (see
363+
`_analyze_long`)."""
268364
if _regex_only(entities, score_threshold):
269365
blank = _BLANK_ARTIFACTS.get(language)
270366
return [
@@ -278,21 +374,29 @@ def _analyze_many(
278374
)
279375
for text in texts
280376
]
281-
return list(
282-
batch_analyzer.analyze_iterator(
283-
texts=texts,
284-
language=language,
285-
batch_size=NLP_BATCH_SIZE,
286-
entities=entities or None,
287-
score_threshold=score_threshold,
288-
ad_hoc_recognizers=ad_hoc_recognizers or None,
289-
)
377+
long_indices = {i for i, text in enumerate(texts) if _exceeds_nlp_length(text, language)}
378+
short_indices = [i for i in range(len(texts)) if i not in long_indices]
379+
results: list[list[RecognizerResult]] = [[] for _ in texts]
380+
short_results = batch_analyzer.analyze_iterator(
381+
texts=[texts[i] for i in short_indices],
382+
language=language,
383+
batch_size=NLP_BATCH_SIZE,
384+
entities=entities or None,
385+
score_threshold=score_threshold,
386+
ad_hoc_recognizers=ad_hoc_recognizers or None,
290387
)
388+
for i, per_text in zip(short_indices, short_results):
389+
results[i] = list(per_text)
390+
for i in sorted(long_indices):
391+
results[i] = _analyze_long(
392+
texts[i], language, entities, score_threshold, ad_hoc_recognizers=ad_hoc_recognizers
393+
)
394+
return results
291395

292396

293397
@asynccontextmanager
294398
async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
295-
global analyzer, batch_analyzer, anonymizer, NER_ENTITIES, _BLANK_ARTIFACTS
399+
global analyzer, batch_analyzer, anonymizer, NER_ENTITIES, _BLANK_ARTIFACTS, _NLP_MAX_LENGTH
296400
logger.info("building analyzer (spacy)")
297401
analyzer = build_analyzer()
298402
batch_analyzer = BatchAnalyzerEngine(analyzer_engine=analyzer)
@@ -319,6 +423,10 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:
319423
language: analyzer.nlp_engine.process_text("", language)
320424
for language in SUPPORTED_LANGUAGES
321425
}
426+
_NLP_MAX_LENGTH = {
427+
language: nlp.max_length
428+
for language, nlp in getattr(analyzer.nlp_engine, "nlp", {}).items()
429+
}
322430
yield
323431

324432

‎apps/sim/lib/guardrails/mask-client.test.ts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,14 @@ describe('maskPIIBatchViaHttp', () => {
7373
expect(mockSleep).toHaveBeenCalledTimes(7)
7474
})
7575

76+
it('stops after three attempts on a persistent 500, which can be a deterministic input failure', async () => {
77+
fetchMock.mockImplementation(async () => new Response('failed', { status: 500 }))
78+
79+
await expect(maskPIIBatchViaHttp(['a'], [])).rejects.toThrow(/mask-batch request failed/)
80+
expect(fetchMock).toHaveBeenCalledTimes(3)
81+
expect(mockSleep).toHaveBeenCalledTimes(2)
82+
})
83+
7684
describe('with a failed-chunk placeholder', () => {
7785
const PLACEHOLDER = '[FAILED]'
7886
const CHUNK_SIZE = 2000

‎apps/sim/lib/guardrails/mask-client.ts‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,15 @@ const CHUNK_CONCURRENCY = env.PII_MASK_CHUNK_CONCURRENCY ?? 64
3636
*/
3737
const MAX_CHUNK_ATTEMPTS = 8
3838

39+
/**
40+
* Attempts for a chunk the route answers with a 500. The route keeps 503 for a
41+
* Presidio outage and 422 for input Presidio refused, so a 500 is an unclassified
42+
* failure that may be deterministic for this input: two retries cover a one-off
43+
* crash without spending the full outage budget resending input that fails the
44+
* same way every time.
45+
*/
46+
const MAX_ATTEMPTS_AFTER_SERVER_ERROR = 3
47+
3948
const RETRYABLE_STATUSES = new Set([408, 429, 500, 502, 503, 504])
4049

4150
class MaskChunkHttpError extends Error {
@@ -196,7 +205,11 @@ async function postChunk(
196205
try {
197206
return await postChunkOnce(url, texts, entityTypes, language, customPatterns)
198207
} catch (error) {
199-
if (attempt >= MAX_CHUNK_ATTEMPTS || !isRetryableChunkError(error)) {
208+
const maxAttempts =
209+
error instanceof MaskChunkHttpError && error.status === 500
210+
? MAX_ATTEMPTS_AFTER_SERVER_ERROR
211+
: MAX_CHUNK_ATTEMPTS
212+
if (attempt >= maxAttempts || !isRetryableChunkError(error)) {
200213
throw error
201214
}
202215
const retryAfterMs = error instanceof MaskChunkHttpError ? error.retryAfterMs : null

‎apps/sim/lib/logs/execution/pii-large-values.test.ts‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -289,6 +289,37 @@ describe('redactManifest — chunk-wise', () => {
289289
})
290290
})
291291

292+
describe('encoded file bodies inside large values', () => {
293+
const encodedFile = Buffer.from(
294+
Array.from({ length: 48 * 1024 }, (_, i) => (i * 7919 + 13) % 256)
295+
).toString('base64')
296+
297+
it('replaces them in the execution log', async () => {
298+
chunkData.set(REF.id, { file: encodedFile, note: 'contact bob' })
299+
300+
const result = await redactLargeValueRefs(
301+
{ finalOutput: REF },
302+
{ entityTypes: ['PERSON'], language: 'en', store: STORE }
303+
)
304+
305+
expect(result.finalOutput).toEqual({
306+
file: '[BINARY_CONTENT_REDACTED]',
307+
note: 'MASKED(contact bob)',
308+
})
309+
})
310+
311+
it('masks them as text in live block state', async () => {
312+
chunkData.set(REF.id, { file: encodedFile })
313+
314+
const result = await redactLargeValueRefsInValue(
315+
{ 'block-1': { output: REF } },
316+
{ entityTypes: ['PERSON'], language: 'en', store: STORE, onFailure: 'throw' }
317+
)
318+
319+
expect(result).toEqual({ 'block-1': { output: { file: `MASKED(${encodedFile})` } } })
320+
})
321+
})
322+
292323
describe('redactLargeValueRefsInValue (arbitrary blockStates)', () => {
293324
it('hydrates + re-stores a ref nested in a non-RedactablePayload shape', async () => {
294325
chunkData.set(REF.id, { note: 'contact bob' })

‎apps/sim/lib/logs/execution/pii-large-values.ts‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,12 @@ export interface RedactLargeValueRefsOptions {
4343
* than feeding a marker (or leaking raw bytes) downstream.
4444
*/
4545
onFailure?: PiiRedactionFailureMode
46+
/**
47+
* @internal Set by {@link redactLargeValueRefs}, the execution log's entry point,
48+
* so encoded file bodies inside its large values are replaced like inline ones
49+
* (see `PiiRedactionOptions.replaceEncodedBinary`). Callers never set it.
50+
*/
51+
replaceEncodedBinary?: boolean
4652
/**
4753
* @internal Shared across every nested invocation spawned from one entry-point
4854
* call, so oversized hydrations serialize globally — not just within their own
@@ -100,7 +106,7 @@ export async function redactLargeValueRefs(
100106
payload: RedactablePayload,
101107
options: RedactLargeValueRefsOptions
102108
): Promise<RedactablePayload> {
103-
const gatedOptions = withOversizedGate(options)
109+
const gatedOptions = withOversizedGate({ ...options, replaceEncodedBinary: true })
104110
// Collect refs across the WHOLE payload first (shared `seen`), so every ref is
105111
// hydrated+masked+re-stored in one bounded-concurrency pass instead of one
106112
// sequential pass per key. A ref shared across keys is walked/masked once.
@@ -269,6 +275,7 @@ async function maskMaterializedValue(
269275
language: options.language,
270276
customPatterns: options.customPatterns,
271277
onFailure: options.onFailure ?? 'scrub',
278+
replaceEncodedBinary: options.replaceEncodedBinary,
272279
})
273280
return redactValueRefs(masked, options)
274281
}

‎apps/sim/lib/logs/execution/pii-redaction.test.ts‎

Lines changed: 66 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import { afterAll, beforeEach, describe, expect, it, vi } from 'vitest'
55
vi.mock('@/lib/auth/internal', () => authInternalMock)
66

77
import {
8+
BINARY_CONTENT_REDACTED_MARKER,
89
PiiRedactionError,
910
REDACTION_FAILED_MARKER,
1011
redactObjectStrings,
@@ -92,7 +93,7 @@ describe('redactPIIFromExecution', () => {
9293
})
9394

9495
it('masks large strings too (never left unredacted)', async () => {
95-
const big = 'x'.repeat(200 * 1024)
96+
const big = 'lorem ipsum '.repeat(17_000)
9697
const payload = { finalOutput: { big, small: 'pii' } }
9798

9899
const result = await redactPIIFromExecution(payload, { entityTypes: [] })
@@ -129,6 +130,70 @@ describe('redactPIIFromExecution', () => {
129130
})
130131
})
131132

133+
/** Base64 of 48KB of non-text bytes, the shape a file body takes in a payload. */
134+
const encodedFile = Buffer.from(
135+
Array.from({ length: 48 * 1024 }, (_, i) => (i * 7919 + 13) % 256)
136+
).toString('base64')
137+
138+
describe('encoded binary content', () => {
139+
it('replaces encoded file bodies in a persisted log without sending them to Presidio', async () => {
140+
const payload = {
141+
traceSpans: [{ blockId: 'write', input: { content: encodedFile, name: 'Ada Lovelace' } }],
142+
workflowInput: { file: `data:application/pdf;base64,${encodedFile}` },
143+
}
144+
145+
const result = await redactPIIFromExecution(payload, { entityTypes: ['PERSON'] })
146+
147+
const [span] = result.traceSpans as { input: { content: string; name: string } }[]
148+
expect(span.input.content).toBe(BINARY_CONTENT_REDACTED_MARKER)
149+
expect((result.workflowInput as { file: string }).file).toBe(BINARY_CONTENT_REDACTED_MARKER)
150+
expect(span.input.name).toBe('MASKED(Ada Lovelace)')
151+
expect(sentTexts()).toEqual(['Ada Lovelace'])
152+
})
153+
154+
it('keeps live data on the masking path, so no string skips redaction by its shape', async () => {
155+
const shapedLikeAFile = `${'Ab1_'.repeat(256)}123-45-6789`
156+
const value = { content: encodedFile, id: shapedLikeAFile }
157+
158+
const result = await redactObjectStrings(value, { entityTypes: [], onFailure: 'throw' })
159+
160+
expect(result).toEqual({ content: `MASKED(${encodedFile})`, id: `MASKED(${shapedLikeAFile})` })
161+
expect(sentTexts()).toEqual([encodedFile, shapedLikeAFile])
162+
})
163+
164+
/** `encodedFile` wrapped at `width` columns, as MIME (76) and PEM (64) encoders emit it. */
165+
const wrapped = (width: number, newline: string) =>
166+
(encodedFile.match(new RegExp(`.{1,${width}}`, 'g')) ?? []).join(newline)
167+
168+
it.each([
169+
['with a trailing newline', `${encodedFile}\n`],
170+
['wrapped at 76 columns', wrapped(76, '\n')],
171+
['wrapped at 64 columns with CRLF', `${wrapped(64, '\r\n')}\r\n`],
172+
['as a wrapped data URL', `data:application/pdf;base64,${wrapped(76, '\n')}`],
173+
['as a data URL with an uppercase scheme', `DATA:application/pdf;base64,${encodedFile}`],
174+
[
175+
'as a data URL with long metadata',
176+
`data:application/pdf;name=${'quarterly-report-'.repeat(40)}.pdf;base64,${encodedFile}`,
177+
],
178+
])('replaces an encoded file body %s in a persisted log', async (_label, body) => {
179+
const result = await redactPIIFromExecution({ finalOutput: body }, { entityTypes: [] })
180+
181+
expect(result.finalOutput).toBe(BINARY_CONTENT_REDACTED_MARKER)
182+
expect(sentTexts()).toEqual([])
183+
})
184+
185+
it.each([
186+
['a newline-separated list of usernames', 'AdaLovelace1815\n'.repeat(100)],
187+
['a short base64 token', 'QWRhIExvdmVsYWNl'],
188+
['a long space-separated sentence', 'Ada Lovelace wrote code in 1843 '.repeat(150)],
189+
])('still masks %s in a persisted log, which is not an encoded file', async (_label, text) => {
190+
const result = await redactPIIFromExecution({ finalOutput: text }, { entityTypes: [] })
191+
192+
expect(result.finalOutput).toBe(`MASKED(${text})`)
193+
expect(sentTexts()).toEqual([text])
194+
})
195+
})
196+
132197
describe('redactObjectStrings', () => {
133198
it('throws PiiRedactionError on masking failure when onFailure is throw', async () => {
134199
maskRoute.mockResolvedValueOnce(rejected())

0 commit comments

Comments
 (0)