feat(kubernetes): create SandboxClaims for matching warm pools

[grs]

Summary

Add support for Kubernetes warm-pool allocation for sandboxes created using a predefined template by creating SandboxClaim resources when a compatible pool exists.

The PR also moves Kubernetes supervisor pod bootstrap and claim activation into the Kubernetes driver path so warm pods can register before assignment and be activated once a claim selects them.

The Kubernetes driver reconciles OpenShell SandboxTemplate resources into Agent Sandbox warm pools when:

• warm pooling is enabled for the Kubernetes driver
• template-backed warm pooling is enabled
• the template has desired_service_level.startup.ready_within
• ready_within is below the configured threshold, defaulting to 5 seconds
• the requested burst size is nonzero, with replicas capped by the driver’s max_replicas

Related Issue

Closes #2157

Changes

• Switched Kubernetes supervisor bootstrap from IssueSandboxToken to the streaming RegisterSupervisor RPC so warm pods can establish a gateway-held registration before they are assigned to a sandbox. This means supervisor path is the same for warm or cold initialisation.
• Added a driver-facing activation interface that keeps supervisor communication routed through the gateway while letting the Kubernetes driver own the claim and pod-specific activation details.
• Moved Kubernetes supervisor pod identity validation into the Kubernetes driver, keeping ServiceAccount token review and pod/Sandbox ownership checks with the Kubernetes-specific code.
• Added Kubernetes warm-pool matching for sandbox create requests using cached SandboxWarmPool and SandboxTemplate fingerprints.
• Created SandboxClaim resources instead of direct Sandbox resources when the sandbox was created with a template and a compatible warm pool exists.
• Added SandboxClaim activation handling that validates the selected Sandbox and pod before issuing a sandbox token.
• Added Kubernetes driver configuration and Helm rendering for enabling or disabling warm pooling.
• Updated RBAC, docs, and tests for the SandboxClaim warm-pool path.

Example

openshell sandbox template create my-template \
  --cpu 0.2 \
  --ready-within 1s \
  --max-burst 1

Create a sandbox from a warmed template:

openshell sandbox create --template my-template -- echo ready

Testing

• mise run pre-commit passes
• Unit tests added/updated
• E2E tests added/updated (if applicable)
• manual test of warm pool matching and warmpool+template reconciliation by kube driver

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rhuss]

Review

Well-engineered PR with sound security design. No critical issues. The registration registry, activation flow, and claim lifecycle are all thoughtfully designed. The K8s-specific code refactor out of the gateway (~856 lines removed from k8s_sa.rs) is a major improvement.

What's done well:

• Auth trust chain is sound. SupervisorBootstrap principal locked to a single RPC, explicitly rejected everywhere else.
• Session-ID concurrency scheme in the registration registry is clean and well-tested.
• Claim creation is idempotent with deterministic naming and preconditioned deletes.
• Level-triggered activation with periodic relist, concurrency caps, and dedup by claim UID.
• Clean abstraction boundary: no K8s types leak through to the gateway.
• Bootstrap security: pod UID cross-checked against SA token extras, ownerReference chain validated, multi-version CRD support.
• Fail-safe behavior throughout: cache not-ready skips allocation, ambiguous matches skip allocation, GC failure creates safe ambiguity.

Inline comments below cover the specific findings.

[rhuss]

naming: RegisterSupervisorPod and PodActivationMessage sit in the driver-agnostic gateway proto. "Pod" leaks K8s terminology into the public API. The internal abstractions already use driver-neutral naming (instance_id, instance_name in SupervisorBootstrapIdentity). Consider renaming to RegisterSupervisor / SupervisorActivationMessage to match the existing ConnectSupervisor pattern. Docker warm containers or pre-booted VMs are plausible future extensions.

[grs]

Resolved

[rhuss]

defaults: Both warmPooling.enabled and profiles.enabled default to true. A fresh install on a cluster without Agent Sandbox CRDs will start watching extension CRDs that may not exist, causing noisy error logs. Consider defaulting to false (opt-in), or at minimum profiles.enabled: false while keeping warm-pool consumption enabled.

[rhuss]

On second thought, this concern is overcautious. Agent Sandbox CRDs are a hard dependency for OpenShell on Kubernetes, so the warm pool resource types will always be present on the cluster. Watching empty resource lists is cheap, and the cache correctly returns NotReady / NoMatch when nothing exists. Defaulting to true is fine here.

[rhuss]

RBAC: create/delete on sandboxclaims is granted unconditionally, even when warmPooling.enabled: false. Contrast with sandboxtemplates/sandboxwarmpools below which correctly gate write verbs on profiles.enabled. Consider gating claim write verbs on warmPooling.enabled.

[grs]

Thanks @rhuss, I have fixed this.

[rhuss]

RBAC scope: When profiles are enabled, create/patch/update/delete on sandboxtemplates and sandboxwarmpools applies cluster-wide. The profile reconciler only operates in one namespace. A namespaced Role would be more minimal. If the cluster-wide scope is intentional (warm pools spanning namespaces), a comment explaining why would help.

[rhuss]

duplicate clients: Both kubernetes_supervisor_bootstrap_identity_provider (line 256) and kubernetes_sandbox_claim_activation_controller (here) call kube::Client::try_default() independently, creating two HTTP connection pools to the apiserver. Consider creating one client and sharing it.

[grs]

Thanks @rhuss, fixed now.

[rhuss]

orphaned records: When the driver returns Unavailable, the gateway preserves the Provisioning record. If the backend create actually failed, this could leak sandbox names in the index. Is there a staleness reconciler for provisioning records, or do they rely on the sandbox watcher to clean up?

[grs]

Thanks @rhuss, also fixed now.

[rhuss]

observability: After exhausting all 3 retry attempts, this returns Status::aborted but doesn't log. A warn! here would provide operational visibility into rapid registration churn.

[grs]

Fixed

[rhuss]

observability: When the JoinSet is at ACTIVATION_MAX_CONCURRENCY (32), the claim event is silently dropped, relying on the 15-second resync to retry. A debug! log here would help operators diagnose delayed activations under load.

[grs]

Fixed

[rhuss]

consistency: All three example ConfigMaps use workspace = "openshell", but E2E tests and docs reference use workspace = "default". Users copying these examples verbatim will create warm pools that don't match sandbox creates in the default workspace. Either align the examples with the default workspace name, or add a comment explaining the workspace must match the operational workspace.

[grs]

These have been removed as the warm pool reconciliation now works of the templates as defined through the openshell api.

[craig-kindo]

Hey, I reviewed this and found that it's not setting the networkPolicyManagement field of the SandboxTemplate for the warm pool. According to Agent Sandbox docs, an unset value is the same as Managed, which has the controller author a NetworkPolicy
over the warm pool pods. This is both (a) different from the cold path, and (b) potentially broken with OpenShell: the default's egress excludes RFC1918, so it would block sandbox supervisors connecting to the gateway if the service ClusterIP is in that range. It also blocks CoreDNS.

The simplest fix is to set networkPolicyManagement to Unmanaged, which opts out of the feature entirely, and change the relevant bits in the template fingerprinting.

[varshaprasad96]

I have a question on this approach (pl feel free to correct me if I'm wrong). So as of now in the flow: warm-pool pod -> sandbox binding has no gw side verification or no restriction on who can trigger it. In the sense, in warm-pool path the gw does not decide which sandbox a pod becomes (it is taken from the pool, and is decided by the k8s driver from the cluster state). The gateway mints the token of trust based on the selected pod from the driver. The gateway does independently verify which pod it's talking to (TokenReview -> pod UID, matched to the registered stream). What it doesn't verify is that this pod is entitled to the sandbox_id the driver supplied.

This means, anyone who can create a sandbox CR (with any sandbox-id label), plus a SandboxClaim gets the trusted agent-sandbox controller to adopt a warm pod for it, and the gateway then mints a JWT for that sandbox_id. The gateway's cluster-wide watch on sandboxclaims means a claim created in any namespace, by anyone is honored.

There is no VAP, or restriction on who can create SandboxClaim or Sandbox (ideally it should only be the gateway). I'm not sure if enforcing that would be the right thing to do in the cluster, but is there any other way we could avoid this problem?

Probably the gateway should only honor a binding for an object it created (e.g., validate the CR UID against a gateway-side record, or mint the sandbox_id as a signed value it can verify), rather than trusting a label anyone can set

[grs]

Thanks @varshaprasad96 !

The assumption prior to this PR is that creation or mutation of Sandbox CRs in the gateway-managed namespace(s) is restricted to trusted parties. The assumption with this PR is that this restriction is extended to SandboxClaim, SandboxTemplate, and SandboxWarmPool. Additionally, TokenReview ensures that only pods running under the configured sandbox ServiceAccount are eligible for further checking. While
it would certainly be possible to strengthen the current validation to cover threats arising from loosening these assumptions, I believe that would be better handled as a separate PR covering both direct and
warm paths.

[varshaprasad96]

sandbox_id silent fallback masks disagreement between the Sandbox CR label and the SandboxClaim

.or_else(...) only fires when the CR label is None. So when the CR label and claim.sandbox_id are both present but differ, there is no comparison and no error — the CR label silently wins and the claim's value is discarded.

This is asymmetric with the cold path, which already treats this class of disagreement as a hard error: validate_sandbox_owner_binding in bootstrap.rs rejects when the pod annotation does not match the live Sandbox CR label (if actual_sandbox_id != sandbox_id { return Err(...) }). A two-source disagreement here is exactly the fingerprint of a planted/forged object, a stale or reused claim, or a controller bug — and it is masked rather than surfaced.

There is test coverage for the CR-label case and the claim-fallback case (activation_request_uses_claim_sandbox_id_when_sandbox_lacks_label), but none for the both-present-and-different case — the code has no branch for it.

Suggested fix: when both sources are present, require equality and error otherwise (mirror the cold path).

let cr_label = sandbox_cr
    .metadata
    .labels
    .as_ref()
    .and_then(|labels| labels.get(LABEL_SANDBOX_ID))
    .filter(|id| !id.is_empty())
    .cloned();

let sandbox_id = match (cr_label, claim.sandbox_id.clone()) {
    (Some(a), Some(b)) if a != b => {
        return Err(format!(
            "sandbox id mismatch: Sandbox CR label {a} != SandboxClaim {b}"
        ));
    }
    (Some(a), _) => a,
    (None, Some(b)) => b,
    (None, None) => {
        return Err(
            "SandboxClaim and selected Sandbox are missing OpenShell sandbox id label".to_string(),
        );
    }
};

[grs]

Thanks @varshaprasad96! Well spotted, this is indeed a bug. I'll push a fix.

[grs]

Hey, I reviewed this and found that it's not setting the networkPolicyManagement field of the SandboxTemplate for the warm pool. According to Agent Sandbox docs, an unset value is the same as Managed, which has the controller author a NetworkPolicy over the warm pool pods. This is both (a) different from the cold path, and (b) potentially broken with OpenShell: the default's egress excludes RFC1918, so it would block sandbox supervisors connecting to the gateway if the service ClusterIP is in that range. It also blocks CoreDNS.

The simplest fix is to set networkPolicyManagement to Unmanaged, which opts out of the feature entirely, and change the relevant bits in the template fingerprinting.

Thanks @craig-kindo! You are right, I missed that but will fix it and include the fix in an upcoming push.

[pimlock]

/ok to test 6f5dc3c

[github-actions]

This pull request has had no activity for 14 days and is now marked stale. It may be closed in 7 days if there is no further activity.

[vvoronko]

/remove-lifecycle stale

[vvoronko]

Two kata/gvisor-relevant gaps in the warm pool path

1. Claim lifecycle: zombie VMs

The claim creation in sandbox_claim_to_k8s_object sets only warmPoolRef with no lifecycle field. Claims default to ShutdownPolicy: Retain — the claimed sandbox and its backing pod/VM are never cleaned up.

For runc this is a minor resource leak (idle containers). For kata, each zombie is a full QEMU VM holding ~250m CPU + 350Mi RAM indefinitely. For gvisor, each zombie holds a runsc sentry process. Under sustained load, resources accumulate until the node is exhausted.

We hit this in agent-sandbox burst testing and documented the root cause in #1306:

• TTLSecondsAfterFinished does not work here — it requires the sandbox to reach a terminal state, but with RestartPolicy: Always (the default), the container restarts indefinitely and TTL never fires
• ShutdownTime (absolute wall-clock deadline) is the correct mechanism — it terminates the sandbox regardless of pod state

Suggested fix: set ShutdownPolicy: Delete + ShutdownTime on claims based on the session's expected duration or a configurable max lifetime. If no duration is known, ShutdownPolicy: Delete alone is still better than Retain.

Data from kata-qemu longevity testing (pool=16, 2-node cluster):

• Without lifecycle: 11+ VMs stuck per node after minutes
• With Delete + ShutdownTime: 138 claims over 10 minutes, zero zombie accumulation

2. Image volume fallback for sandboxed runtimes

PR #1300 switched supervisor sideload to ImageVolumeSource (KEP-4639) by default. However, OCI image volumes are not supported by kata (kata-containers#13749) or gvisor (gvisor#14471). The warm pool path uses the same global supervisor_sideload_method config with no per-runtime override.

This means kata/gvisor warm pool pods will silently fail to mount the supervisor binary. The only workaround today is setting supervisor_sideload_method: init-container globally, which downgrades all sandboxes (including runc) to the slower init-container path.

Suggested fix: when rendering a warm pool template whose runtimeClassName is a sandboxed runtime (kata-*, gvisor), automatically fall back to init-container sideload regardless of the global setting. This is the same pattern #1300 introduced for cold-path pods, just needs to be runtime-aware.

[vvoronko]

Kata/gVisor runtime compatibility report for warm pool integration

We cross-validated OpenShell's runtime-dependent features against the kata-containers and gVisor runtime stacks to identify gaps that could affect warm pool deployments. Here's the full matrix:

#	Feature	Kata status	gVisor status	Notes
1	ImageVolume sideload	N/A — use initContainer	N/A — use initContainer	initContainer method (driver.rs:4658-4720) works under both runtimes (emptyDir backed by virtiofs/9p in kata). Warm pools should configure supervisor_sideload_method: init-container for kata/gVisor pools. Per-runtime override requested below.
2	Landlock filesystem policy	Works (kata 4.0+)	Unsupported, degrades safely	Kata guest kernel has CONFIG_SECURITY_LANDLOCK=y since 1487eaaaa (kata 4.0.0+). OpenShell's BestEffort mode (default) logs a High-severity OCSF alert and continues without Landlock on older kata or gVisor.
3	Network init iptables	Eliminated by proxy-pod	Eliminated by proxy-pod	proxy-pod topology (#2885) moves network enforcement to Kubernetes NetworkPolicy (CNI-level), completely outside the VM/sentry boundary. No iptables inside the kata guest.
4	Shared PID /proc inspection	Eliminated by proxy-pod	Eliminated by proxy-pod	proxy-pod doesn't use shareProcessNamespace. No cross-runtime /proc divergence.
5	AppArmor	Silently stripped by kata runtime-rs	Limited in gVisor	Irrelevant under proxy-pod — all caps dropped, no AppArmor needed for enforcement.
6	GPU passthrough	Requires VFIO config	Experimental (nvproxy)	Not warm-pool-specific; existing per-runtime setup applies.

Bottom line: zero blockers for kata or gVisor warm pools, provided:

1. Pools with kata/gVisor RuntimeClass use supervisor_sideload_method: init-container (or a per-runtime override — see below).
2. proxy-pod topology (feat(kubernetes): add proxy-pod topology (in-pod process supervisor, out-of-pod proxy) #2885) is used for kata deployments (eliminates iptables, caps, and shared-PID-namespace concerns).

Actionable request: The global supervisor_sideload_method setting has no per-runtime override. When a gateway serves pools with mixed RuntimeClass (e.g., runc + kata), the operator must choose one method globally. ImageVolume is faster for runc but unsupported by kata (kata#13749) and gVisor (gvisor#14471). Consider falling back to initContainer when the pool's RuntimeClass is kata or gVisor, as implemented in #1300 for non-pool sandboxes.

[vvoronko]

/ok-to-test

[grs]

@vvoronko I have pushed an update that includes setting the ShutdownPolicy to Delete. The need for aligment between supervisor_sideload_method (currently global config only) and the runtime class is indeed an important issue, but I feel that it should be a separate PR. Is there an issue open for that specific issue yet?

[vvoronko]

@grs not in OpenShell, there is an issue opened for agent-sandbox bad default
I agree that supervisor_sideload_method could be in additional PR, should I open an issue for that on OpenShell?