Skip to content

fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest - #698

Open
deanq wants to merge 5 commits into
mainfrom
deanq/sls-994-manifest-env
Open

deanq wants to merge 5 commits into
mainfrom
deanq/sls-994-manifest-env

Conversation

@deanq

@deanq deanq commented Nov 7, 2026

Copy link
Copy Markdown
Member

What

rp flash deploy wrote every resource's env into the build manifest, including literal values such as env={"HF_TOKEN": os.environ["HF_TOKEN"]}. That manifest is sent to the backend when the build is finalized, and also packed into the artifact as runpod_manifest.json. Neither copy needs the endpoint values.

Changes:

  • Queue and API env is still checked against reserved names, then left out of the manifest. Endpoints already get their env from the endpoint template, so workers see the same values.
  • Task env stays in the manifest sent to the backend, because the backend launches deployed task pods from it.
  • The packaged runpod_manifest.json has no env for any resource. Nothing reads env from that file.
  • Deploy warns when an env name ending in _TOKEN, _KEY, _SECRET or _PASSWORD has a literal value, and points to rp secret add plus runpod.Secret(...). The warning names the variable, never the value.

Why this shape

runpod.Secret values were already sent as references, so only literal values were affected. Literal values still have to reach the endpoint template or task definition, which is where they're used. The warning steers secret-looking values to runpod.Secret, the only way to keep a value out of those definitions.

Test plan

  • New tests in tests/test_apps/test_deploy.py:
    • queue and API env is not in the manifest, and the literal value appears nowhere in it
    • task env stays in the manifest
    • reserved endpoint env is still rejected
    • the packaged manifest has no env, and the caller's manifest is left unchanged
    • secret-looking literal names warn without logging the value
    • runpod.Secret values and ordinary names don't warn
  • The env, packaging and warning tests failed before the change
  • make quality-check (2783 passed)
  • Live check: deployed a queue endpoint with a literal DEMO_TOKEN and MODE. The CLI printed the warning, the worker returned both values, and the stored build manifest did not contain the literal value. A deploy of the same app from unfixed code stored the value in the manifest.

Deploy wrote every resource's env, literal values included, into the
manifest it uploads and packs into the build artifact.

- Queue and API env is checked against reserved names, then left out
  of the manifest. Endpoints get their env from the endpoint template.
- Task env stays in the manifest sent to the backend, which launches
  deployed task pods from it.
- The packaged runpod_manifest.json carries no env for any resource.
- Warn when a secret-looking env name (*_TOKEN, *_KEY, *_SECRET,
  *_PASSWORD) has a literal value, pointing to runpod.Secret.

SLS-994
@celine-runpod celine-runpod changed the title fix(apps): keep env values out of the build manifest fix(apps): keep env values out of the build manifest (SLS-994) Nov 9, 2026
@celine-runpod celine-runpod changed the title fix(apps): keep env values out of the build manifest (SLS-994) fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest (SLS-994) Nov 9, 2026
@celine-runpod

Copy link
Copy Markdown
Contributor

bugbot run

@deanq
deanq requested a balanced review from Copilot November 9, 2026 18:02
@deanq deanq changed the title fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest (SLS-994) fix(apps): user secrets (e.g. HF_TOKEN) are written in plain text into the uploaded build manifest Nov 9, 2026

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the changes aimed at reducing literal-secret exposure in deployment manifests.

No open issues 🎉

What changed in this PR

This PR reduces exposure of literal environment secrets in deployment manifests while preserving how workers receive their environment values.

Changes:

  • Removes queue and API environment values from uploaded manifests, retaining reserved-name validation and task environment values needed by the backend.
  • Strips all resource environment values from the packaged manifest without modifying the caller’s manifest.
  • Warns about secret-looking literal environment values without logging their contents, with regression tests for filtering and warnings.
File Description
tests/​test_apps/​test_deploy.py Adds coverage for environment filtering, task retention, reserved names, packaging, and secret warnings.
runpod/​apps/​deploy.py Filters manifest environment values and warns about secret-looking literals.

Review effort: Balanced

Comment thread runpod/apps/deploy.py Resolved
deanq and others added 2 commits November 9, 2026 11:19
Endpoint env was serialized into the manifest entry and then popped back
out. Build the entry without it instead, via a new include_env flag on
ResourceSpec.to_manifest, and validate the reserved keys from the spec.
Behavior is unchanged: endpoints carry no env in the manifest, task env
still goes to the backend.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants