feat(kubernetes): create SandboxClaims for matching warm pools by grs · Pull Request #2460 · NVIDIA/OpenShell

grs · 2026-07-24T09:48:24Z

Summary

Add transparent Kubernetes warm-pool allocation by matching sandbox create requests to OpenShell-enabled SandboxWarmPool templates and creating SandboxClaim resources when a compatible pool exists. The PR
also moves Kubernetes supervisor pod bootstrap and claim activation into the Kubernetes driver path so warm pods can register before assignment and be activated once a claim selects them.

Related Issue

Closes #2157

Changes

Switched Kubernetes supervisor bootstrap from IssueSandboxToken to the streaming RegisterSupervisorPod RPC so warm pods can establish a gateway-held registration before they are assigned to a sandbox. This means supervisor path is the same for warm or cold initialisation.
Added a driver-facing activation interface that keeps supervisor communication routed through the gateway while letting the Kubernetes driver own the claim and pod-specific activation details.
Moved Kubernetes supervisor pod identity validation into the Kubernetes driver, keeping ServiceAccount token review and pod/Sandbox ownership checks with the Kubernetes-specific code.
Prevent spoofing in activation by requiring an opaque token registered on sandbox creation request.
Added Kubernetes warm-pool matching for sandbox create requests using cached SandboxWarmPool and SandboxTemplate fingerprints.
Created SandboxClaim resources instead of direct Sandbox resources when exactly one compatible warm pool exists.
Added SandboxClaim activation handling that validates the selected Sandbox and pod before issuing a sandbox token.
Added Kubernetes driver configuration and Helm rendering for enabling or disabling warm pooling.
Added reconciler in Kubernetes driver that will create SandboxWarmPool and associated SandboxTemplate in response to presence of labeled ConfigMap with toml describing the sandbox parameters it should match. Included some example ConfigMaps.
Updated RBAC, docs, and tests for the SandboxClaim warm-pool path.

Examples

The easiest way to try it out is to use one (or more) of the examples under examples/kubernetes-warm-pool-config/: default-configmap.yaml will create a warm pool that will be matched by sandbox create requests with no explicit arguments, env-foo-configmap.yaml by requests with --env FOO=bar and cpu-0-2-configmap.yaml by requests with --cpu 0.2. Obviously you can also modify these for other settings. The examples set the pool size to 1; if you want to test rapid creation of multiple sandboxes you will need to increase that.

Testing

mise run pre-commit passes
Unit tests added/updated
E2E tests added/updated (if applicable)
manual test of warm pool matching and warmpool+template reconciliation by kube driver

Checklist

Follows Conventional Commits
Commits are signed off (DCO)
Architecture docs updated (if applicable)

copy-pr-bot · 2026-07-24T09:48:27Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

Signed-off-by: Gordon Sim <gsim@redhat.com>

…eation Signed-off-by: Gordon Sim <gsim@redhat.com>

Signed-off-by: Gordon Sim <gsim@redhat.com>

rhuss

Review

Well-engineered PR with sound security design. No critical issues. The registration registry, activation flow, and claim lifecycle are all thoughtfully designed. The K8s-specific code refactor out of the gateway (~856 lines removed from k8s_sa.rs) is a major improvement.

What's done well:

Auth trust chain is sound. SupervisorBootstrap principal locked to a single RPC, explicitly rejected everywhere else.
Session-ID concurrency scheme in the registration registry is clean and well-tested.
Claim creation is idempotent with deterministic naming and preconditioned deletes.
Level-triggered activation with periodic relist, concurrency caps, and dedup by claim UID.
Clean abstraction boundary: no K8s types leak through to the gateway.
Bootstrap security: pod UID cross-checked against SA token extras, ownerReference chain validated, multi-version CRD support.
Fail-safe behavior throughout: cache not-ready skips allocation, ambiguous matches skip allocation, GC failure creates safe ambiguity.

Inline comments below cover the specific findings.

rhuss · 2026-07-30T15:41:11Z

+  // The initial trial activates already-bound cold pods immediately. Later
+  // warm-pool stages keep this stream pending until a SandboxClaim adopts the
+  // registered pod.
+  rpc RegisterSupervisorPod(RegisterSupervisorPodRequest)


naming: RegisterSupervisorPod and PodActivationMessage sit in the driver-agnostic gateway proto. "Pod" leaks K8s terminology into the public API. The internal abstractions already use driver-neutral naming (instance_id, instance_name in SupervisorBootstrapIdentity). Consider renaming to RegisterSupervisor / SupervisorActivationMessage to match the existing ConnectSupervisor pattern. Docker warm containers or pre-booted VMs are plausible future extensions.

rhuss · 2026-07-30T15:41:11Z

+  # v1beta1 SandboxClaim resources when a compatible OpenShell-enabled
+  # SandboxWarmPool exists in the target namespace.
+  warmPooling:
+    enabled: true


defaults: Both warmPooling.enabled and profiles.enabled default to true. A fresh install on a cluster without Agent Sandbox CRDs will start watching extension CRDs that may not exist, causing noisy error logs. Consider defaulting to false (opt-in), or at minimum profiles.enabled: false while keeping warm-pool consumption enabled.

rhuss · 2026-07-30T15:41:11Z

+  - apiGroups:
+      - extensions.agents.x-k8s.io
+    resources:
+      - sandboxclaims


RBAC: create/delete on sandboxclaims is granted unconditionally, even when warmPooling.enabled: false. Contrast with sandboxtemplates/sandboxwarmpools below which correctly gate write verbs on profiles.enabled. Consider gating claim write verbs on warmPooling.enabled.

rhuss · 2026-07-30T15:41:11Z

+  - apiGroups:
+      - extensions.agents.x-k8s.io
+    resources:
+      - sandboxtemplates


RBAC scope: When profiles are enabled, create/patch/update/delete on sandboxtemplates and sandboxwarmpools applies cluster-wide. The profile reconciler only operates in one namespace. A namespaced Role would be more minimal. If the cluster-wide scope is intentional (warm pools spanning namespaces), a comment explaining why would help.

rhuss · 2026-07-30T15:41:11Z

+) -> Option<Arc<SandboxClaimActivationController>> {
+    std::env::var_os("KUBERNETES_SERVICE_HOST")?;
+
+    match kube::Client::try_default().await {


duplicate clients: Both kubernetes_supervisor_bootstrap_identity_provider (line 256) and kubernetes_sandbox_claim_activation_controller (here) call kube::Client::try_default() independently, creating two HTTP connection pools to the apiserver. Consider creating one client and sharing it.

rhuss · 2026-07-30T15:41:11Z

                self.sandbox_index.remove_sandbox(sandbox.object_id());
                Err(Status::failed_precondition(status.message().to_string()))
            }
+            Err(status) if status.code() == Code::Unavailable => {


orphaned records: When the driver returns Unavailable, the gateway preserves the Provisioning record. If the backend create actually failed, this could leak sandbox names in the index. Is there a staleness reconciler for provisioning records, or do they rely on the sandbox watcher to clean up?

rhuss · 2026-07-30T15:41:11Z

+        }
+    }
+
+    Err(Status::aborted(


observability: After exhausting all 3 retry attempts, this returns Status::aborted but doesn't log. A warn! here would provide operational visibility into rapid registration churn.

rhuss · 2026-07-30T15:41:11Z

+    activator: Arc<dyn SupervisorBootstrapActivator>,
+    claim: DynamicObject,
+) {
+    if tasks.len() >= ACTIVATION_MAX_CONCURRENCY {


observability: When the JoinSet is at ACTIVATION_MAX_CONCURRENCY (32), the claim event is silently dropped, relying on the 15-second resync to retry. A debug! log here would help operators diagnose delayed activations under load.

rhuss · 2026-07-30T15:41:11Z

+data:
+  warm-pool.toml: |
+    version = 1
+    workspace = "openshell"


consistency: All three example ConfigMaps use workspace = "openshell", but E2E tests and docs reference use workspace = "default". Users copying these examples verbatim will create warm pools that don't match sandbox creates in the default workspace. Either align the examples with the default workspace name, or add a comment explaining the workspace must match the operational workspace.

grs requested review from a team, derekwaynecarr, maxamillion and mrunalp as code owners July 24, 2026 09:48

grs marked this pull request as draft July 24, 2026 09:49

grs force-pushed the pod-registration branch from 148f85f to efbce1f Compare July 24, 2026 10:19

grs mentioned this pull request Jul 24, 2026

refactor(server): decouple K8s ServiceAccount bootstrap from selected compute driver config #2023

Open

5 tasks

grs force-pushed the pod-registration branch 3 times, most recently from cdfe4be to 45f2384 Compare July 29, 2026 11:35

grs added 15 commits July 29, 2026 12:38

fix(python): make proto generation install grpcio tools

84be163

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(k8s): bootstrap supervisors through pod registration stream

98e00bf

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(server): add warm supervisor pod registration groundwork

1b91b34

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(server): scaffold claim-driven warm pod activation

514169f

Signed-off-by: Gordon Sim <gsim@redhat.com>

refactor(kubernetes): move supervisor bootstrap identity into driver

16b906f

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(kubernetes): activate warm pods from SandboxClaims

2f30b66

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(kubernetes): create SandboxClaims for matching warm pools

5aa9e00

Signed-off-by: Gordon Sim <gsim@redhat.com>

feat(kubernetes): add support for automatic warm pool and template cr…

eee939c

…eation Signed-off-by: Gordon Sim <gsim@redhat.com>

test(kubernetes): add warm pool e2e coverage

fdef4d1

Signed-off-by: Gordon Sim <gsim@redhat.com>

fix(kubernetes): reconcile ambiguous warm claim creation

94b29d6

Signed-off-by: Gordon Sim <gsim@redhat.com>

fix(kubernetes): reconcile missed warm claim activations

c6e902a

Signed-off-by: Gordon Sim <gsim@redhat.com>

fix(server): expire activated pod registration tombstones

de7aba7

Signed-off-by: Gordon Sim <gsim@redhat.com>

fix(kubernetes): reactivate restarted warm pods

dab0d88

Signed-off-by: Gordon Sim <gsim@redhat.com>

fix(kubernetes): gate warm pool profiles behind HA lease

6872156

Signed-off-by: Gordon Sim <gsim@redhat.com>

refactor(kubernetes): centralize workspace namespace mapping

7c11c99

Signed-off-by: Gordon Sim <gsim@redhat.com>

grs force-pushed the pod-registration branch from 45f2384 to 7c11c99 Compare July 29, 2026 11:59

grs marked this pull request as ready for review July 29, 2026 17:28

matthewgrossman mentioned this pull request Jul 29, 2026

feat: warmpool support for OpenShell sandboxes #1447

Closed

rhuss mentioned this pull request Jul 30, 2026

Switch supervisor->gateway communication to event-based streaming over supervisor sessions #1731

Open

rhuss reviewed Jul 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(kubernetes): create SandboxClaims for matching warm pools - #2460

feat(kubernetes): create SandboxClaims for matching warm pools#2460
grs wants to merge 15 commits into
NVIDIA:mainfrom
grs:pod-registration

grs commented Jul 24, 2026 •

edited

Loading

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

rhuss left a comment

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

rhuss Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

grs commented Jul 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related Issue

Changes

Examples

Testing

Checklist

Uh oh!

copy-pr-bot Bot commented Jul 24, 2026

Uh oh!

rhuss left a comment

Choose a reason for hiding this comment

Review

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

grs commented Jul 24, 2026 •

edited

Loading