Skip to content

lk agent deploy --image-tar silently ignores --deployment and updates production #971

Description

@tmtcam-042

Summary

Running a prebuilt-image deployment with --deployment staging updates the agent’s production deployment. The CLI exits successfully and prints “Deployed agent”, with no warning that the requested deployment was ignored.

This makes the command unsafe for staging workflows in CI.

Environment

  • LiveKit CLI v2.18.0 on Linux.
  • Prebuilt container uploads enabled for the project.
  • Also inspected v2.18.6 source, which has the same missing deployment propagation. The stock v2.18.6 binary was not tested.

Reproduction

Warning: this reproduction can update production. Use a disposable agent.

  1. Create an agent with a running production deployment and record its current version.

  2. From the directory containing that agent’s livekit.toml, run:

    lk agent deploy \
      --deployment staging \
      --image-tar ./image.tar
  3. Inspect the agent’s deployment status and version history.

Expected behaviour

Deploy the image to staging and leave production unchanged.

If prebuilt uploads do not support named deployments, reject the command before uploading or modifying the agent.

Actual behaviour

  • The CLI exits with code 0 and prints “Deployed agent”.
  • Production advances to a new build version.
  • No staging deployment appears.

We reproduced this using the identical image already running in production, verified that the image digest was unchanged, and rolled back to the previous production version afterwards. No secrets were changed.

Source investigation

In deployAgent, the --image/--image-tar branch calls deployPrebuiltImage and returns before the function reads --deployment.

The prebuilt helper calls GetPushTarget(ctx, agentID). That SDK method sends only agent_id to /push-target. The source-build path instead passes the deployment selector to DeployAgentV2.

Relevant code:
https://github.com/livekit/livekit-cli/blob/v2.18.6/cmd/lk/agent.go#L831-L883

Additional experiment

We patched the CLI and SDK locally to forward deployment=staging as a query parameter to /push-target. A live same-image upload still updated production.

This establishes that forwarding that query parameter alone is insufficient. We cannot determine from the public code whether the upload service needs an implementation change or expects a different mechanism for selecting deployments.

Requested resolution

  • Immediately reject unsupported combinations of named deployments and prebuilt uploads, rather than silently targeting production.
  • Support the deployment selector throughout the prebuilt upload path, or document the supported mechanism.
  • Add regression coverage proving that a staging upload leaves production unchanged.

Related issue

#860 reported another silently ignored option on the prebuilt-image path, affecting secret updates:
#860

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions