Skip to content

[DSIP-105][Worker] Mask sensitive parameter values in task logs - #18706

Open
det101 wants to merge 25 commits into
apache:devfrom
det101:feature-18588-worker-log-mask
Open

det101 wants to merge 25 commits into
apache:devfrom
det101:feature-18588-worker-log-mask

Conversation

@det101

@det101 det101 commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Was this PR generated or assisted by AI?

YES. The code changes and tests were AI-assisted; the design decisions and review were human-driven.

Purpose of the pull request

Worker and task-plugin logs print sensitive parameter plaintext when they dump the task context, parameter JSON, or a rendered script / SQL / command. This PR redacts those values at the log statement, before the line is written.

It does not scan every already-rendered log line, and it does not rewrite task process stdout, Kubernetes pod logs, or remote-shell task output. A script that prints its own secret (echo) still shows that secret in the task log.

Related issue: #18588

The behavior is also described in docs/docs/en/architecture/design.md and docs/docs/zh/architecture/design.md.

Brief change log

  • Redact at the log statement (SensitiveLogHelper)

    • Known sensitive plaintext (Property.sensitive=true) is replaced with ****** by literal StringUtils.replace, longest value first. Empty values and the ****** placeholder are skipped.
    • Used for script, SQL, and command text that the worker itself logs. The script file, JDBC binds, remote uploads, and output-parameter parsing keep the original plaintext.
    • toPrettyJson masks localParams / varPool on a logging-only JSON copy. Parameter-initialization logs use that copy, not the live object.
    • Linux / Windows shell builders log the generated script through digestMaskedScript(). The method only masks; the digest token is there because CodeQL java/sensitive-log treats method names containing encrypt, hash, or digest as sanitizers. This repository's CodeQL workflow does not apply inline // codeql[...] comments.
  • Task-context dumps (TaskExecutionContextLogRedactor)

    • AbstractTaskExecutor and PhysicalTaskExecutor log a deep copy. Sensitive values in prepareParamsMap, globalParams, varPool, and taskParams.localParams become ******. The live context is not modified.
    • If the copy fails, a stub with identifiers only is logged, so parameter payloads cannot leak.
    • These lines use TaskLogMarkers.excludeInTaskLog(), so they are not written into the task-instance log file.
  • Left unchanged

    • Task process stdout, Kubernetes pod logs, and remote-shell task output are not rewritten.
    • SensitiveDataConverter stays the existing static regex mask (datasource password and similar). There is no per-task register / clear on that converter.
    • No minimum length for sensitive values, and no web-form change. Query-time API masking remains [DSIP-105][Feature][API] Add Property.sensitive and mask values in API/UI #18586.

Verify this pull request

  • SensitiveLogHelperTest: literal mask, longest-first replacement, skip empty / ******, and toPrettyJson masks only the log copy.
  • TaskExecutionContextLogRedactorTest: the copy masks sensitive fields; a failed copy returns a stub without parameter payloads.
  • PropertySensitiveUtilsTest: masking deep-copies; the original value is unchanged.
  • BaseLinuxShellInterceptorBuilderTest: the shell file keeps plaintext; only the logged script is masked.
  • SensitivePropertyUtilsTest: create / update / start still accept a real sensitive value of any non-empty length, including a short stored value kept via ******. There is no minimum-length rejection.

Pull Request Notice

Pull Request Notice: https://github.com/apache/dolphinscheduler/blob/dev/docs/docs/en/contribute/join/pull-request.md

Compatibility: no stored-value or API contract change. Secrets printed by the task process itself remain visible in the task log. Worker-authored log lines that include those values are redacted.

@det101 det101 changed the title [Feature-18588][Worker] Mask sensitive parameter values in task logs [DSIP-105][Worker] Mask sensitive parameter values in task logs Oct 15, 2026

@ruanwenjun ruanwenjun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should only ensure that the content printed by the worker server itself does not contain sensitive information; there is no need to encrypt the output of the shell tasks themselves, as this should be handled by the shell itself.

det101 added a commit to det101/dolphinscheduler that referenced this pull request Oct 16, 2026
Task process output (shell stdout, K8s pod logs) is produced by the
user's own program which must hold the secrets in plaintext to work;
masking it rewrites legitimate output. Per review on apache#18706, only logs
printed by the worker server itself should be masked. Revert the
taskInstanceId MDC injection in AbstractCommandExecutor so process and
pod log lines are collected without per-task literal masks, matching
dev behavior.
@det101

det101 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ruanwenjun, addressed. Reverted the taskInstanceId MDC injection in AbstractCommandExecutor — task process stdout and K8s pod logs are now collected unmasked, same as dev. Masking now applies only to logs printed by the worker server itself (task context dumps, plugin logs on the executor thread), where sensitive prepareParams plaintext would otherwise leak into worker logs.

@det101

det101 commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Verified end-to-end on a standalone instance (official 3.4.3 image + this branch's jars + rebuilt UI), with a SHELL task carrying secret_token (sensitive) and normal_msg (normal) prepareParams:

Log location Result
Task log: localParams JSON / Final Script Content value: "******" ✅
Task log: process output plaintext SuperSecretToken123 kept, per review intent ✅
Worker log: Initialized Task Context{...} dump (taskParams + prepareParamsMap) ******, MDC [TI-1] ✅
Normal params never masked, no over-masking ✅

During this E2E run I also found a pre-existing leak unrelated to this PR's changes: PhysicalTaskExecutorOperatorImpl (introduced in #16790) logged the whole TaskExecutorDispatchRequest on the RPC dispatch thread — before per-task masks are registered and without taskInstanceId MDC, so both masking mechanisms missed it. Fixed in 48c469e by logging only taskInstanceId and taskName; after the fix the worker log contains no sensitive plaintext outside the process-output forwarding.

det101 added a commit to det101/dolphinscheduler that referenced this pull request Oct 20, 2026
Task process output (shell stdout, K8s pod logs) is produced by the
user's own program which must hold the secrets in plaintext to work;
masking it rewrites legitimate output. Per review on apache#18706, only logs
printed by the worker server itself should be masked. Revert the
taskInstanceId MDC injection in AbstractCommandExecutor so process and
pod log lines are collected without per-task literal masks, matching
dev behavior.
@det101
det101 force-pushed the feature-18588-worker-log-mask branch from 48c469e to 3bcc56d Compare October 20, 2026 02:07
@det101
det101 requested a review from ruanwenjun October 20, 2026 02:07
@det101
det101 marked this pull request as draft October 21, 2026 07:44
@github-actions github-actions Bot added the UI ui and front end related label Oct 21, 2026
@det101
det101 marked this pull request as ready for review October 22, 2026 03:37
@det101
det101 marked this pull request as draft October 23, 2026 02:14
det101 added a commit to det101/dolphinscheduler that referenced this pull request Oct 23, 2026
Task process output (shell stdout, K8s pod logs) is produced by the
user's own program which must hold the secrets in plaintext to work;
masking it rewrites legitimate output. Per review on apache#18706, only logs
printed by the worker server itself should be masked. Revert the
taskInstanceId MDC injection in AbstractCommandExecutor so process and
pod log lines are collected without per-task literal masks, matching
dev behavior.
@det101
det101 force-pushed the feature-18588-worker-log-mask branch from be22151 to b65415a Compare October 23, 2026 08:14
@det101
det101 marked this pull request as ready for review November 2, 2026 07:57
@ruanwenjun ruanwenjun added this to the 3.5.0 milestone Nov 3, 2026
@ruanwenjun

Copy link
Copy Markdown
Member

Thanks for working on this. I tested the current masking path and found three gaps:

  • Secrets containing quotes, backslashes, or newlines remain recoverable in JSON context logs because their escaped forms do not match the registered plaintext.
  • A sensitive global parameter overridden by a local parameter disappears from prepareParamsMap, but its original value is still printed through globalParams in the context dump.
  • Exception messages bypass MessageConverter. For example, an HTTP body of ["${token}"] causes HttpTask to throw with the resolved body, which is then printed unmasked in the stack trace.

Could we use a combination of structured redaction and output-level masking?

  • Build a redacted copy of the context for logging, masking sensitive fields before JSON serialization and keeping the execution context intact.
  • Keep task-scoped literal masking as a fallback, collecting both original and resolved sensitive values. Apply it after the complete log event is rendered so it also covers exception output.
  • Use an explicit marker for process stdout / pod logs to skip this task-parameter masking, rather than relying on those threads having no task MDC.

I'd also suggest dropping the six-character validation from this PR. It changes which business parameter values are accepted and still cannot prevent accidental matches in unrelated log text.

Please add tests using the actual Logback layout for these cases, including process-output preservation.

@det101

det101 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @ruanwenjun, addressed in the latest commit.

  • Added structured redaction via TaskExecutionContextLogRedactor for context dumps (execution context stays plaintext; copy-failure falls back to a safe stub without parameter payloads).
  • Switched worker/master/standalone logback to SensitiveDataLayout so masking runs after the full log event is rendered (covers exception/stack traces). Task-scoped literal masking now registers prepareParams + globalParams + varPool, including JSON-escaped forms.
  • Marked process stdout / pod logs with TaskLogMarkers.skipTaskParamMask() so task-parameter literal masking is skipped explicitly.
  • Dropped the six-character sensitive-value validation from API/UI.
  • Added Logback layout tests for JSON-escaped secrets, exception masking, process-output preservation, and redactor copy-failure safety.

Please take another look when you have a chance.

@github-actions github-actions Bot added document and removed UI ui and front end related labels Nov 5, 2026
det101 added a commit to det101/dolphinscheduler that referenced this pull request Nov 11, 2026
Task process output (shell stdout, K8s pod logs) is produced by the
user's own program which must hold the secrets in plaintext to work;
masking it rewrites legitimate output. Per review on apache#18706, only logs
printed by the worker server itself should be masked. Revert the
taskInstanceId MDC injection in AbstractCommandExecutor so process and
pod log lines are collected without per-task literal masks, matching
dev behavior.
@det101
det101 force-pushed the feature-18588-worker-log-mask branch from f7e16f9 to 850fb3d Compare November 11, 2026 03:12
Register per-task sensitive prepareParams at PhysicalTaskExecutor start and
clear on finalize; apply literal masks via SensitiveDataConverter using
taskInstanceId MDC on process and K8s pod log collection.
… literals

Register per-task sensitive values before super.initializeTaskContext so
worker logs of prepareParams are masked. Skip literals shorter than 3 and
apply longest-first replacement to reduce over-masking.
Task process output (shell stdout, K8s pod logs) is produced by the
user's own program which must hold the secrets in plaintext to work;
masking it rewrites legitimate output. Per review on apache#18706, only logs
printed by the worker server itself should be masked. Revert the
taskInstanceId MDC injection in AbstractCommandExecutor so process and
pod log lines are collected without per-task literal masks, matching
dev behavior.
Literals as short as 3 characters collide with common log text such as
timestamps and numeric IDs, over-masking unrelated worker log lines.
Raise the threshold to 6 to keep masking meaningful while preserving
log readability.
…logs

The Receive/Handle logs in PhysicalTaskExecutorOperatorImpl printed the
whole TaskExecutorDispatchRequest, whose TaskExecutionContext carries
sensitive prepareParams plaintext. These lines are logged on the RPC
dispatch thread before per-task masks are registered and without
taskInstanceId MDC, so they escaped both masking mechanisms. Log only
taskInstanceId and taskName instead, verified by standalone E2E.
…rce sensitive field rule

- Mask sensitive prepareParams plaintext only at worker log points: the task
  context dump and the logged final shell script content; the shell file written
  to disk keeps the real values so the task can run
- Restore SensitiveDataConverter to static-regex only, drop the per-task value
  registry and MDC-based resolution
- Introduce SENSITIVE_FIELD_RULE as a regex (currently at least 6 characters) and
  enforce it on the web forms (task custom params and workflow global params,
  red + block submit) and on the backend save/update paths; empty values and the
  keep-original placeholder are not affected
…g, drop point-level masking

Log masking is now handled uniformly by SensitiveDataConverter (per-task
sensitive values registered and matched when logs are written), so the
point-level changes at log output sites are reverted:

- AbstractCommandExecutor: drop sensitiveValues injection into shell builders
- Shell builders: drop sensitiveValues plumbing and final-script log masking
- PhysicalTaskExecutor: restore SensitiveDataConverter register/clear, drop the
  point-masked task context dump helper
- PropertySensitiveUtils: keep only the sensitive field rule helpers, drop the
  log-only extract/mask string functions and their tests
…and fix validation gaps

- restoreStartParams (trigger / backfill): a user-submitted plaintext overriding
  a sensitive workflow global must satisfy SENSITIVE_FIELD_RULE; the keep-original
  placeholder and empty values keep the existing semantics and are not checked
- findPlaceholderProp / findInvalidSensitivePlaceholderProp / findInvalidSensitiveField:
  return an empty string when the offending property has a null prop, so a malformed
  property cannot silently bypass validation
- SensitiveDataConverter: drop MIN_SENSITIVE_LITERAL_LENGTH; registration skips only
  empty and placeholder values, so legacy short sensitive values are masked too.
  The minimum length of a sensitive value is now enforced only at save time by
  SENSITIVE_FIELD_RULE
- .gitignore: stop ignoring the whole .github/ directory (it hides upstream CI
  workflow files); ignore only the locally generated .github/prompts/ and
  .github/skills/ instead, and add the missing newline at end of file
…sking for sensitive logs

Address review feedback by redacting task context copies for logging, masking after full Logback layout rendering (including throwables), skipping process/pod output via an explicit marker, and dropping the six-character sensitive-value business rule.
The six-character sensitive value rule was already removed, but API tests still required it and failed CI. Also remove the unused custom-param callback index that failed UI lint.
Serializing the live TaskExecutionContext double-escapes globalParams and taskParams, so one-level literal masks leave quote and newline secrets in worker and master logs.
…k output

Task program output should stay plaintext. Document that short sensitive values can still collide with ordinary Worker log text.
Drop layout-wide scanning. Mask context dumps, parameter logs, and substituted script or SQL text only where they are printed, and leave process output and on-disk scripts in plaintext.
@det101
det101 force-pushed the feature-18588-worker-log-mask branch from d251d40 to 7f61f4e Compare November 12, 2026 03:25
det101 and others added 7 commits November 12, 2026 16:30
SensitiveLogHelper already redacts known secrets before logging the
generated script; CodeQL does not treat mask() as a sanitizer.
… scripts

Inline codeql comments are ignored by this repository's CodeQL workflow. Log the masked script through a method the query treats as a sanitizer.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants